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I.  INTRODUCTION 

A.         BACKGROUND 

When  contractors  submit  proposals  in  response  to  a  Government  Request  for 
Proposals  (RFP),  their  proposals  must  be  evaluated  to  determine  which  one  represents  the 
best  value  to  the  Government.  The  evaluation  factors  on  which  the  proposals  are  scored  can 
vary  in  number  and  type.  Cost,  past  performance,  and  quality  must  be  included  as  factors. 
When  evaluating  the  cost  factor,  the  Procuring  Contracting  Officer  (PCO),  must  evaluate  the 
contractor's  cost  estimates  of  the  product  or  service. 

Cost  estimating  systems  that  produce  reliable  price  proposals  are  a  key  safeguard  to 
obtaining  fair  and  reasonable  contract  prices  (GAO  95-3, 1995:p.  11).  When  a  contract  price 
is  higher  because  it  is  based  on  inaccurate,  incomplete,  and  non-current  data,  the  Government 
has  the  right  to  a  price  reduction  under  the  provisions  of  the  Truth  in  Negotiations  Act 
(TINA).  When  contracts  are  overpriced  because  of  poor  estimating,  the  Government  may 
not  recover  the  overstated  price  under  TINA. 

Government  Administrative  Contracting  Officers  (ACOs),  with  help  from  auditors 
in  the  Defense  Contract  Audit  Agency  (DCAA),  are  responsible  for  determining  the 
adequacy  of  contractors'  estimating  systems.  When  the  DCAA  reports  an  estimating 
deficiency,  DoD  regulations  establish  procedures  and  time  frames  for  their  correction.  If  a 
contractor  does  not  make  adequate  progress  in  correcting  the  deficiency,  ACOs  are 
authorized  to  take  actions  to  obtain  correction,  such  as  reducing  or  suspending  progress 


payments,  recommending  nonaward  of  potential  contracts,  or  disapproving  all  or  part  of  the 
estimating  system  (DFARS,  1996:  215-81 1-70). 

The  process  of  formal  disapproval  is  a  series  of  contractor's  written  responses  to 
DC  AA  audit  findings,  corrective  action  plans,  and  ACO  evaluations.  This  process  can  take 
years  before  a  final  determination  is  made  (Rich  Buhre  interview).  The  final  notice  of 
disapproval  must  be  sent  to  each  Contracting  Officer  and  Contract  Administrative  Office 
having  substantial  business  with  the  contractor. 

Once  notified,  PCOs  negotiating  proposals  generated  by  an  estimating  system  with 
an  identified  deficiency  shall  evaluate  whether  the  deficiency  impacts  the  negotiations.  If 
it  does,  the  PCO  has  several  alternatives  provided  in  the  Defense  Federal  Acquisition 
Regulation  Supplement  (DFARS): 

(1)  Allowing  the  contractor  additional  time  to  correct  the  estimating  system 
deficiency  and  submit  a  corrected  proposal. 

(2)  Considering  another  type  of  contract. 

(3)  Using  additional  cost  analysis  techniques  to  determine  the  reasonableness 
of  the  cost  elements  affected  by  the  system's  deficiency. 

(4)  Segregating  the  questionable  areas  as  a  cost  reimbursable  line  item. 

(5)  Reducing  the  negotiation  objective  for  profit  or  fee. 

(6)  Including  a  contract  (reopener)  clause  that  provides  for  adjustment  of  the 
contract  amount  after  an  award.  (DFARS,  1996:215.81 1) 

Numerous  General  Accounting  Office  (GAO)  reports  have  identified  poor  contractor 
cost  estimating  systems  as  a  significant  problem  in  Government  contracting.     These 


inadequate  systems  have  potentially  cost  the  Government  $3.67  billion  between  the  years 
1987  and  1991  (GAO  95-3,  1995:  p.  10). 

B.  OBJECTIVES  OF  THE  RESEARCH 

The  area  of  Contractor  Cost  Estimating  Systems  is  far  too  large  for  a  single  thesis. 
Thus,  this  thesis  will  focus  on  how  PCOs  are  informed  of  estimating  system  disapprovals  and 
deficiencies  and  what  actions  they  might  take  after  this  information  is  obtained.  The 
objectives  of  this  research  effort  will  be  from  the  Government's  perspective,  including: 

1 .  How  PCOs  obtain  information  of  poor  contractor  cost  estimating  systems. 

2.  How  this  information  is  used  once  obtained. 

C.  RESEARCH  QUESTION 

In  order  to  accomplish  the  objectives  of  this  research  effort,  the  following  primary 
research  question  is  addressed: 

What  action  does  the  Procuring  Contracting  Officer  take  when  notified  that  a 
contractor's  estimating  system  has  deficiencies  or  has  been  disapproved  by  the 
Administrative  Contracting  Officer? 

The  following  subsidiary  questions  were  applicable  to  this  research  effort: 

1.  Are  the  alternative  actions  specified  in  DFARS  215.811 -70(g)(2)  adequate  to 
motivate  a  contractor  to  respond  to  the  PCO's  desires? 

2.  Are  there  more  effective  actions  currently  being  used  by  PCOs  and  if  so 
should  they  be  incorporated  into  the  DFARS? 


3.    If  the  PCO  gathers  independent  data  indicating  a  contractor  has  estimating 
system  deficiencies,  how  does  this  affect  the  performance  of  his  or  her  duties? 


D.  SCOPE  OF  THE  RESEARCH 

This  study  focuses  on  the  collection  methods  and  usage  of  information  on  contractor 
cost  estimating  systems  by  PCOs.  Analysis  is  limited  to  these  two  areas  to  maintain  a 
manageable  research  topic  size.  To  collect  data  for  analysis,  a  survey  was  conducted  of 
various  procurement  programs  and  projects  with  Research,  Development,  Testing  and 
Evaluation  (RDT&E),  budgets  of  more  than  $25  million  throughout  DoD.  Additionally, 
interviews  have  been  conducted  with  DCAA,  Defense  Contract  Management  Command 
(DCMC),  and  GAO  personnel.  The  use  of  activities  nationwide  within  DoD  provides  a 
broad  statistical  base  for  Government  representation.  Contact  via  correspondence  provided 
sufficient  input.  By  using  questionnaires  to  gain  data  for  analysis,  the  researcher  assumed 
that  respondees  were  answering  truthfully  and  in  full. 

E.  RESEARCH  METHODOLOGY 

The  researcher  used  three  methods  of  data  collection  to  complete  the  this  effort. 
First,  a  comprehensive  review  of  the  available  literature  was  conducted  utilizing  the  Naval 
Postgraduate  School  (Knox)  Library,  the  Systems  Management  Acquisition  Library,  the 
Defense  Logistics  Studies  Information  Exchange  (DLSIE),  and  the  Defense  Technical 
Information  Center  (DTIC).  Next,  surveys  were  sent  to  selected  contracting  personnel  at 
procurement  programs  and  projects  with  RDT&E  budgets  of  more  than  $25  million. 


Surveys  were  developed  through  research  using  previously  discussed  sources.  Lastly, 
telephone  interviews  were  also  conducted  on  a  limited  basis  to  clarify  and  enhance  responses 
to  the  survey. 

F.         ORGANIZATION 

The  research  has  been  divided  into  six  chapters.  In  this  Chapter,  the  objectives  of  the 
research  have  been  set  forth,  the  scope  and  direction  of  the  effort  identified,  and  the 
methodologies  for  data  collection  presented. 

Chapter  II  discusses  the  regulatory  environment  applicable  to  cost  estimating 
systems. 

Chapter  III  explores  cost  estimating  systems  and  summarizes  a  review  of  the 
literature. 

Chapter  IV  presents  the  survey  questionnaires  sent  to  the  various  activities,  the 
responses  received,  and  an  analysis  of  the  data. 

Chapter  V  presents  the  principal  findings,  conclusions,  and  recommendations 
generated  by  this  research. 


II.  COST  ESTIMATING  REGULATORY  ENVIRONMENT 

A.         INTRODUCTION 

DoD  annually  awards  billions  of  dollars  in  contracts  for  a  wide  variety  of  products 
and  services.  Many  of  these  procurements  are  conducted  in  the  absence  of  the  competitive 
marketplace  that  establishes  fair  and  reasonable  contract  prices.  Therefore,  DoD  Contracting 
Officers  rely  on  price  proposals  developed  and  submitted  by  contractors  to  a  great  extent. 
Price  proposals  are  developed  by  contractors  from  their  cost  estimating  systems.  Therefore, 
sound  estimating  systems  are  fundamental  in  determining  fair  and  reasonable  contract  prices. 

Recognizing  the  Government's  vulnerability  in  negotiating  noncompetitive  contract 
prices,  the  Congress  passed  the  Truth-in-NegotiationsAct  (TINA)  in  1 962  (GAO  93-1,1 992: 
p.  1 ).  The  Act,  intended  to  protect  the  Government  against  inflated  cost  estimates,  required 
contractors  to  submit  cost  or  pricing  data  to  support  price  proposals  and  to  certify  that  the 
data  submitted  were  current,  accurate,  and  complete.  By  requiring  certified  data,  TINA  tries 
to  put  the  Government  on  equal  informational  footing  with  contractors. 

To  insure  that  contractors  are  submitting  data  that  are  current,  accurate,  and  complete, 

the  Federal  Acquisition  Regulation  (FAR)  establishes  contractor  estimating  systems  review 

requirements.  It  directs  that: 

Cognizant  audit  activities,  when  it  is  appropriate  to  do  so,  shall  establish  and 
manage  regular  programs  for  reviewing  selected  contractors'  estimating 
systems  or  methods,  in  order  to  (1)  reduce  the  scope  of  reviews  to  be 
performed  on  individual  proposals,  (2)  expedite  the  negotiation  process,  and 
(3)  increase  the  reliability  of  proposals  (FAR,  1996:  15.81 1). 


The  auditor  is  to  send  a  copy  of  the  survey  report  to  each  PCO  and  ACO  having  substantial 
business  with  that  contractor  (FAR,  1996:  15.81 1).  When  the  auditor  is  determining  the 
acceptability  of  a  contractor's  estimating  system,  he  or  she  should  consider: 


1 .  The  source  of  data  for  estimates  and  the  procedures  for  ensuring  that  the 
data  are  accurate,  complete,  and  current. 

2.  The  documentation  developed  and  maintained  in  support  of  the  estimate. 

3.  The  assignment  of  responsibilities  for  originating,  reviewing,  and 
approving  estimates. 

4.  The  procedures  followed  for  developing  estimates  for  direct  and  indirect 
cost  elements. 

5.  The  extent  of  coordination  and  communication  between  organizational 
elements  responsible  for  the  estimate. 

6.  Management  support,  including  estimate  approval,  establishment  of 
controls,  and  training  programs.  (FAR:  15.81 1) 


B.         DEFENSE    FEDERAL    ACQUISITION    REGULATION    SUPPLEMENT 
CONCERNING  ESTIMATING  SYSTEMS 

The  DFARS  defines  DoD  policy  in  regards  to  contractors'  estimating  systems  by 

establishing  applicability,  responsibilities,  and  characteristics  of  an  adequate  estimating 

system.  This  regulation  applies  to  all  DoD  contractors  classified  as  large.  To  be  considered 

a  large  contractor,  the  company  has  to  meet  one  of  the  two  following  criteria: 

1 .  In  its  preceding  fiscal  year  the  contractor  received  DoD  prime  contracts 
or  subcontracts  totaling  $50  million  or  more  for  which  certified  cost  or 
pricing  data  were  required. 


2.  If  in  its  preceding  fiscal  year  the  contractor  received  DoD  prime  contracts 
or  subcontracts  totaling  $10  million  or  more  (but  less  than  $50  million)  for 
which  certified  cost  or  pricing  data  were  required  and  the  contracting  officer, 
with  concurrence  of  the  ACO,  determines  it  to  be  in  the  best  interest  of  the 
Government  (e.g.,  significant  estimating  problems  are  believed  to  exist  or  the 
contractor's  sales  are  predominantly  Government).  (DFARS,  1996:  215.81 1) 


The  responsibilities  of  the  PCO,  ACO,  Auditor,  and  Contractor  classified  as  a  large 
business  are  spelled  out  in  the  DFARS  in  more  specific  terms  than  the  FAR.  The 
responsibility  of  the  PCO  under  the  DFARS  is  to  apply  the  disclosure,  maintenance,  and 
review  requirements  (Cost  Estimating  Systems  Requirement  clause)  to  large  businesses 
(DFARS,  1996:  215.81 1-70).  The  ACO  is  to  determine  the  adequacy  of  the  system  and 
pursue  the  correction  of  any  deficiencies.  Auditors,  on  behalf  of  the  ACO,  are  to  act  as  team 
leaders  in  conducting  the  systems  reviews.  Lastly,  the  Contractor  subject  to  this  requirement 
shall  do  the  following: 

( 1 )  Maintain  an  adequate  system. 

(2)  Describe  its  system  to  the  ACO. 

(3)  Provide  timely  notice  of  changes  in  the  system. 

(4)  Correct  system  deficiencies  identified  by  the  ACO.  (DFARS,  1996: 
215.811-70)  ' 

DFARS  also  provides  characteristics  of  adequate  estimating  systems.     These 

characteristics  should  be  considered  when  the  ACO  is  evaluating  the  contractor's  estimating 

system.  These  characteristics  are  as  follows: 

1 .  Establishes  clear  responsibility  for  preparation,  review  and  approval  of 
cost  estimates. 

2.  Provides  a  written  description  of  the  organization  and  duties  of  the 
personnel  responsible  for  preparing,  reviewing  and  approving  cost  estimates. 


3.  Assures  that  relevant  personnel  have  sufficient  training,  experience  and 
guidance  to  perform  estimating  tasks  in  accordance  with  the  contractor's 
established  procedures. 

4.  Identifies  the  sources  of  data  and  the  estimating  methods  and  rationale 
used  in  developing  cost  estimating. 

5.  Provides  for  appropriate  supervision  throughout  the  estimating  process. 

6.  Provides  for  consistent  application  of  estimating  techniques. 

7.  Provides  for  detection  and  timely  correction  of  errors. 

8.  Protects  against  cost  duplication  and  omissions. 

9.  Provides  for  the  use  of  historical  experience,  including  a  historical  vendor, 
pricing  information,  where  appropriate. 

10.  Requires  use  of  appropriate  analytical  methods. 

1 1 .  Integrates  information  available  from  other  management  systems,  where 
appropriate. 

1 2.  Requires  management  review  including  verification  that  the  company's 
estimating  policies,  procedures  and  practices  comply  with  this  regulation. 

1 3 .  Provides  for  internal  review  of  and  accountability  for  the  adequacy  of  the 
estimating  system,  including  the  comparison  of  projected  results  to  actual 
results  and  an  analysis  of  any  differences. 

14.  Provides  procedures  to  update  cost  estimates  in  a  timely  manner 
throughout  the  negotiation  process. 

15.  Addresses  responsibility  for  review  and  analysis  of  the  reasonableness 
of  subcontract  prices.  (DFARS,  1996:  215.811-70) 

C.         DEFENSE  LOGISTICS  AGENCY  DIRECTIVES 

Defense  Logistics  Agency  Directive  (DLAD)  5000.4  defines  specifically  the  duties 
of  the  ACO  within  the  DCMC  in  regards  to  estimating  system  deficiencies.  The  ACO  is 
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responsible  for  five  actions  including:  preparing  an  outstanding  estimating  system  deficiency 
memorandum,  reviewing  the  contractor's  corrective  action  plan,  monitoring  the  contractor's 
corrective  action,  verifying  correction  of  deficiencies,  and  notifying  the  contractor  of  system 
adequacy  (DLAD  5000.4,  1995:  chap.  2). 

When  the  ACO  determines  that  the  contractor's  estimating  system  contains 
deficiencies  that  require  correction  based  on  an  estimating  systems  survey,  the  ACO  will 
prepare  a  memorandum  that  does  the  following: 

1 .  Lists  the  cost  elements  covered. 

2.  Identifies  all  deficiencies  (significant  or  otherwise)  that  require  correction 
(including  date  initially  reported). 

3.  Addresses  the  contractor's  progress  in  correcting  the  deficiencies  against 
the  milestones  in  its  corrective  action  plan. 

4.  Estimates  the  fiscal  risk  to  the  Government  resulting  from  the 
deficiencies. 

5.  Discusses  action  taken  during  the  analysis  and  negotiation  of  individual 
proposals  impacted  by  the  deficiency  to  ensure  that  fair  and  reasonable  prices 
are  negotiated. 

6.  Documents  steps  taken  by  the  ACO  to  ensure  that  the  contractor  corrects 
the  deficiencies  in  a  timely  manner. 

7.  Forwards  a  copy  of  the  DCAA  audit  report  and  other  pertinent  documents 
and  correspondence.  (DLAD  5000.4,  1995:  chap.  2) 

This  memorandum  will  be  prepared  and  forwarded  to  the  DCMC  District  estimating  system 
focal  point.  The  memorandum  will  be  updated  at  least  semiannually  to  reflect  the 
contractor's  progress  (or  lack  thereof).  Updated  memoranda  are  due  to  the  district  estimating 
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system  focal  point  on  1  February  and  1  August  of  each  year.  The  primary  purpose  of  the 
memorandum  is  to  keep  the  District  staff  informed  of  the  situation  and  confident  that  the 
ACO  has  taken  appropriate  action  to  remedy  the  situation.  It  also  provides  DCMC  insight 
as  to  the  most  common  type  of  deficiency  and  the  cost  elements  most  often  affected.  The 
memorandum  may  also  be  forwarded  to  buying  offices  or  used  as  an  enclosure  to  field 
pricing  reports  (however,  this  action  does  not  eliminate  the  need  for  the  impact  of  the 
deficiency  on  the  particular  proposal  to  be  quantified  within  any  particular  pricing  report). 
(DLAD  5000.4,  1995:  chap.  2) 

The  ACO,  with  help  from  DCAA  auditors  and  other  specialists,  will  evaluate  the 
contractor's  proposed  corrective  actions  for  adequacy  and  completeness.  This  evaluation 
should  include  associated  milestone  dates  for  reasonableness  and  timeliness.  The  ACO  will 
apprise  the  contractor  of  the  adequacy  of  the  plan  in  writing.  (DLAD  5000.4,  1995:  chap.  2) 
The  ACO  has  a  continuing  responsibility  to  monitor  the  contractor's  progress  in  correcting 
deficiencies.  If  the  contractor  fails  to  make  adequate  progress  against  the  established 
milestone  dates,  the  contractor's  estimating  system  should  be  disapproved  in  part  or  in  whole 
by  the  ACO.  The  ACO  shall  take  whatever  action  is  necessary  to  ensure  that  the  contractor 
corrects  the  deficiencies.  (DLAD  5000.4,  1995:  chap.  2) 

Upon  receipt  of  notification  from  the  contractor  that  the  deficiencies  have  been 
corrected,  the  ACO  will  verify  that  the  deficiencies  have  been  eliminated.  When  the 
deficiencies  have  been  eliminated,  the  contractor  should  be  notified  that  its  estimating  system 
is  considered  adequate  (and  if  applicable,  withdrawal  of  estimating  system  disapproval). 
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This  notice  will  also  be  sent  to  the  DCAA  auditor,  and  the  PCOs  doing  substantial  business 
with  the  contractor.  (DLAD  5000.4,  1995:  chap.2) 

D.         DEFENSE  CONTRACT  AUDIT  AGENCY  PROCEDURES 

DCAA's  general  audit  policy  is  that  the  auditor  is  responsible  for  evaluating 
contractor  estimating  methods  and  procedures.  This  is  accomplished  by  estimating  system 
surveys  designed  to  evaluate  the  reliability  of  a  contractor's  estimates,  to  identify  areas 
requiring  special  emphasis  in  the  audit  and  negotiation  of  individual  price  proposals,  and  to 
recommend  improvements.  Estimating  system  surveys  provide  knowledge  of  the  strengths 
and  weaknesses  in  a  contractor's  estimating  system.  (CAM,  1995:  p.  51 18)  It  is  DoD  policy 
that  contractor  estimating  systems  are  reviewed  every  three  years  unless  past  experience  and 
current  audit  risk  is  considered  to  be  low  (DFARS,  1995:  215.81 1-70). 

In  addition  to  meeting  the  DFARS  requirement  for  review,  an  estimating  system 
survey  serves  as  the  fundamental  basis  for  determining  audit  risk  and  scope  for  future  price 
proposal  and  defective  pricing  audits  within  DCAA.  The  primary  objectives  of  the  survey 
are  to  accomplish  the  following: 

1.  Evaluate  the  adequacy  of  a  contractor's  system  for  developing  cost 
estimates  for  price  proposal  purposes. 

2.  Evaluate  a  contractor' s  compliance  with  its  written  estimating  procedures 
and  disclosed  estimating  system  (if  applicable). 

3.  Identify  areas  of  a  contractor's  estimating  system  requiring  special 
emphasis  or  attention  during  the  audit  and  negotiation  of  individual  price 
proposals. 
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4.  Inform  interested  government  activities  of  the  reliability  of  a  contractor's 
estimating  system,  and  of  actions  necessary  to  correct  existing  deficiencies. 
(CAM,  1995:  p.  5120) 


The  DCAA  estimating  systems  survey  is  to  examine  the  following  areas: 

1.  Internal  Audits.  The  contractor's  estimating  system  should  be  subjected  to 
periodic  internal  audits  to  ensure  that  the  company's  policies  and  procedures  are  being 
followed  and  comply  with  applicable  rules  and  regulations,  and  that  prior  audit  findings  have 
been  corrected.  The  objective  is  to  recognize  the  contractor's  commitment  to  self- 
governance.  Contractors  may  comply  with  this  standard  by  using  in-house  efforts  or 
through  the  services  of  outside  consultants.  (CAM,  1995:  p.  5124) 

2.  System  Description.  The  contractor  should  provide  a  written  description  of  its 
estimating  system  in  enough  detail  to  allow  the  auditor  to  get  a  thorough  understanding  of 
how  the  system  operates.  This  write-up  should  include  the  overall  organization,  assignment 
of  responsibilities,  and  flow  of  information  between  each  process  and  function  within  the 
system  as  well  as  other  management  system.  (CAM,  1995:  p.  5124) 

3.  Training.  Training  should  be  provided  to  appropriate  employees  on  Government 
regulations  and  company  policies  and  procedures.  It  should  include  use  of  statistical  aids 
and  advanced  estimating  techniques.  (CAM,  1995:  p.  5127) 

4.  Cost  Estimate  Development.  A  major  part  of  the  survey  is  an  evaluation  of  the 
effectiveness  of  the  methods  and  procedures  used  to  develop  estimates  of  individual  cost 
elements.  A  contractor  should  ensure  that  estimates  of  source  data  are  appropriately  applied 
and  the  basis  for  estimating  each  cost  element  is  provided  by  written  guidelines  for 

14 


developing  consistent  proposals.  Procedures  for  documenting  proposals  should  cover  the 
assumptions  used,  the  basis  of  each  cost  element,  and  the  milestones  for  contract 
performance.  (CAM,  1995:  p.  5127) 

5.  Estimating  Methods.  The  appropriateness  of  estimating  methods  used  by  the 
contractor  in  preparing  proposals  should  be  evaluated.  Auditors  should  examine  the  degree 
that  factors  such  as  the  nature  of  the  product  or  service,  the  firmness  of  specifications,  and 
the  prior  experience  with  related  products  or  services  have  been  taken  into  account  in  the 
proposals.  A  contractor's  estimating  system  should  clearly  identify  its  pricing  policy.  The 
estimating  systems  survey  should  examine  the  extent  that  estimating  methods  make 
appropriate  use  of  historical  cost  data.  Analytical  methods  should  be  used,  when  appropriate 
to  develop  cost  estimates  or  evaluate  the  reasonableness  of  cost  estimates  developed  using 
other  procedures. 

In  the  area  of  spare  parts  and  contract  changes,  the  reasonableness  of  formula  pricing 
methods  are  to  be  reviewed  to  ensure  individual  elements  of  costs  are  not  duplicated  in  base 
cost  and  loading  factors.  Additionally,  the  suitability  of  catalog  pricing  and  prepriced  listing 
methods  for  developing  reasonable  prices  for  spare  parts  proposals  needs  to  be  evaluated. 
Acceptability  of  the  contractor's  methods  for  developing  cost  estimates  for  contract  changes 
must  be  assessed.  (CAM,  1995:  p.  5128) 

6.  Proposal  Format  and  Support.  Proposal  format  and  support  should  be  evaluated 
to  determine  that  proposals  are  submitted  on  appropriate  forms  (or  acceptable  substitutes) 
and  all  data  required  are  furnished.    An  appraisal  of  detail  contained  in  proposals  and 
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supporting  data  will  be  conducted.  In  addition,  a  review  of  the  adequacy  of  contractor's 
identification  of  cost  and  pricing  data  submitted  in  support  of  the  proposal  and  the  related 
Pricing  Certification  should  also  be  examined.  (CAM,  1995:  p.  5129) 

7.  Subcontract  Price/Cost  Analysis.  The  contractor's  policies  and  procedures  should 
require  that  a  price  analysis  be  performed  for  all  subcontracts  that  require  submission  of  cost 
or  pricing  data.  When  the  contractor  is  unable  to  perform  an  adequate  price  analysis,  a  cost 
analysis  should  be  submitted.  The  results  of  such  analyses  should  be  included  as  part  of  the 
contractor' s  cost/price  analysis  and  are  critical  to  the  negotiation  of  fair  and  reasonable  prime 
contract  prices.  For  this  reason,  contractors  should  have  policies  and  procedures  in  place  to 
accomplish  such  analyses  prior  to  the  submission  of  their  own  cost  or  pricing  data.  (CAM, 
1995:  p.  5129) 

8.  Control  and  Review  of  Estimates.  Survey  the  contractor's  estimating  controls  and 
reviews  to  determine  the  sufficiency  of  internal  controls.  This  review  of  controls  is  to  ensure 
uniformity  of  approach,  timely  detection  and  correction  of  errors,  and  prevention  of  cost 
duplications  and  omissions.  The  contractor's  level  of  supervision  throughout  the  estimating 
process  impacts  the  soundness  of  judgmental  estimates  and  adherence  to  established 
procedures.  The  quality,  frequency,  scope,  and  results  of  management  reviews  of  the 
estimating  function  should  be  noted.  (CAM,  1995:  p.  5129) 

9.  Contract  Certification.  The  contractor  should  have  established  policies  and 
procedures  for  ensuring  that  all  cost  or  pricing  data  are  current,  accurate,  and  complete  as  of 
the  date  of  agreement  on  price  (CAM,  1995:  p.  5130). 
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E.         SUMMARY 

DoD  has  provided  a  system  of  regulations  to  insure  that  contractors  are  submitting 
proposals  that  will  place  the  contractor  and  the  Government  on  an  equal  informational 
footing.  The  ACO  is  responsible  for  determining  the  adequacy  of  the  contractor's  estimating 
system.  DCAA  auditors  actually  conduct  the  reviews  called  estimating  system  surveys. 
Once  completed,  the  ACO  is  responsible  to  insure  any  deficiencies  are  corrected.  Although 
the  regulations  and  process  is  rather  straightforward,  the  next  Chapter  will  show  that  many 
problems  can  and  do  exist  in  the  cost  estimating  systems  arena. 
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III.  COST  ESTIMATING  SYSTEMS 

A.         OVERVIEW  OF  COST  ESTIMATING 

Cost  estimating  requires  the  application  of  analysis  and  experienced  judgment  in 
projecting  material,  labor,  and  subcontract  requirements  to  complete  a  project  on  time  and 
within  budget.  Timing  constraints  and  the  availability  of  historical  data  have  an  impact  on 
the  quality  of  the  estimate.  Selection  of  appropriate  estimating  techniques  requires  extensive 
analysis  by  the  contractor.  Appropriateness  of  selected  estimating  techniques  should  be 
reviewed  periodically  by  the  contractor.  The  techniques  used  when  the  program  is  in  the 
concept  exploration  stage,  or  when  no  bill  of  materials  exists,  are  usually  not  the  techniques 
that  would  be  appropriate  for  programs  in  the  production  and  fielding  stage.  Because  cost 
estimating  integrates  technical  as  well  as  financial  information,  the  process  requires  input 
from  every  organizational  element  of  the  business.  (CAM,  1995:  p.  D16) 

Although  contractor  estimating  systems  differ  in  approach  and  philosophy,  their  basic 

objectives  are  the  same.  Cost  estimates  are  a  series  of  informed  projections  and  assumptions 

based  on  available  information  existing  at  the  time  of  proposal  preparation.  (CAM,  1995:  p. 

D16)  Cost  estimating  is  series  of  logical  steps.  The  level  of  detail  required  in  each  step  is 

affected  by  the  contract  requirements  expressed  in  the  RFP.  Typical  steps  in  cost  estimating 

include  the  following: 

1 .  Ensuring  that  all  relevant  background  documents  such  as  historical  costs, 
drawings,  and  specifications  are  available  to  assist  in  understanding  job 
requirements. 
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2.  Determining  which  estimating  techniques  will  be  used,  the  level  of  detail 
required,  and  the  amount  of  time  available  to  generate  and  document  a 
completed  estimate. 

3.  Determining  if  quotes  and  other  information  will  be  required  from  outside 
sources. 

4.  Deciding  if  any  elements  require  further  clarification,  redesign,  or  have 
potential  manufacturing  difficulties. 

5.  Determining  if  the  capability  and  capacity  to  manufacture  required 
components  exist  in-house. 

6.  Determining  if  further  information  is  required  to  develop  and  complete 
estimates. 

7.  Coordinating  the  activities  of  departments  participating  in  the  estimating 
exercise. 

8.  Obtaining  quotes,  history,  and  other  bases  for  material  and  subcontract 
items. 

9.  Assembling  direct  costs  by  cost  element,  and  computing  indirect  expenses 
using  appropriate  factors  and  rates. 

1 0.  Consolidating  proposal  elements  and  documenting  preparation  rationale. 
(CAM,  1995:  p.  D 16) 


B.         TYPES  OF  COST  ESTIMATING 

The  types  of  cost  estimating  methods  selected  are  dictated  by  the  availability  of 

historical  data,  the  nature  of  product  or  service,  and  the  Government  requirements.  Cost 

estimating  methods  may  be  categorized  into  six  main  groups:  subjective,  parametric, 

comparative,  synthetic,  global,  and  research  and  development  (R&D): 

1.  Subjective.  This  estimating  method  develops  costs  using 
experience,  judgment,  memory,  informal  notes,  and  other  readily  available 
data.    Typically,  these  kinds  of  estimates  are  used  in  proposals  when  a 


drawing  has  not  yet  been  developed  or  the  contractor  is  faced  with  limited 
proposal  preparation  time. 

2.  Parametric.  This  method  creates  labor  and  material  estimates  by 
statistically  analyzing  and  manipulating  historical  data  to  reflect  current 
quantity  requirements.  For  example,  previous  raw  material  requirements  on 
a  price  per  pound  basis  could  be  used  to  project  current  proposal  amounts. 
Parametrics  use  one  or  more  cost  estimating  relationships  (CER)  to  estimate 
costs  associated  with  the  development,  manufacture,  or  modification  of  an 
end  item.  Special  cost  comparisons  are  required  to  validate  parametric 
estimating  systems.  Variables  used  in  CERs  must  be  logically  related  and 
statistically  valid.  The  rationale  for  selecting  the  variables  should  be  well 
documented.  Parametrics  are  often  used  to  cross-check  estimates  developed 
using  other  estimating  techniques. 

3.  Comparative.  This  method  develops  proposed  costs  using  like 
items  produced  in  the  past  as  a  surrogate.  Allowances  are  made  for  product 
dissimilarities  and  changes  in  complexity,  scale,  design,  and  materials.  The 
comparative  method  can  be  used  in  conjunction  with  parametric  estimating 
and  can  be  used  to  develop  adjusted  unit  costs  while  parametrics  are  applied 
to  project  the  newly  proposed  quantities.  Improvement  curve  applications  are 
an  example  of  comparative  estimating. 

4.  Synthetic.  This  method  divides  proposals  into  their  smallest 
component  tasks.  Estimates  are  developed  for  component  tasks  which  make 
up  the  whole.  Synthetic  estimates  are  normally  supported  by  detailed  bills 
of  material. 

5.  Global.  This  is  a  quick  and  subjective  technique  used  to  determine 
the  advisability  of  continuing  with  a  project. 

6.  Research  and  Development  (R&D).  There  are  two  basic 
approaches  available  for  the  R&D  type  of  estimating.  The  first  is  a  simple 
form  of  targeting  R&D  objectives  in  the  context  of  a  fixed  budget.  As  in  the 
preparation  of  routine  budgets,  the  breakdown  should  be  compatible  with  the 
cost  accounting  system  and  procedures  established  to  monitor  and  control 
expenditures.  A  second  method  of  estimating  R&D  is  a  trial  and  error 
procedure  involving  an  interchange  of  ideas  and  information  including  all 
available  records  of  past  R&D  effort  and  experience.  Because  there  are  so 
many  unknown  factors  involved  in  R&D  effort,  the  potential  for  error  in  this 
type  of  estimating  is  especially  great.  (CAM,  1995:  p.  D18) 
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C.         ESTIMATING  PROCESS  AT  A  TYPICAL  CONTRACTOR 

Although  each  contractor  has  their  own  estimating  system  that  fits  their  business,  the 
following  is  an  example  of  a  typical  contractor.  Generally  at  large  contractors,  the  estimating 
(or  pricing)  department  usually  has  overall  responsibility  for  coordinating  and  assembling 
estimates  to  be  incorporated  into  proposals  authorized  by  top  management.  Preparation  of 
detailed  estimates  is  accomplished  by  the  departments  that  will  actually  perform  or  supervise 
the  work  if  the  contract  is  received.  (CAM,  1995:  p.  D16) 

The  cost  estimating  venture  is  usually  initiated  in  response  to  an  RFP.  The  RFP 
provides  a  statement  of  work  (SOW)  that  outlines  the  Government's  requirements.  It  is  also 
a  source  of  information  in  establishing  a  baseline  for  labor  and  material  requirements.  The 
contractor's  proposal  should  include  tasks  and  materials  consistent  with  the  SOW.  When 
top  management  authorizes  a  response  to  an  RFP,  the  estimating  department  reviews  the 
SOW  and  any  top-management  guidance  and  issues  a  cost  estimate  request  to  other 
departments  within  the  company  that  will  be  involved  in  the  proposal.  The  estimating 
department  generally  has  primary  responsibility  for  coordinating  the  overall  submission  and 
authorizing  the  finalized  proposal.  (CAM,  1995:  p.  D16) 

Contractors  may  also  submit  unsolicited  proposals  that  have  no  outstanding  RFPs. 
When  such  proposals  are  pursued  by  a  buying  activity,  the  PCO  will  normally  request  a  more 
detailed  cost  proposal  before  requesting  an  audit.  The  estimating  process  should  be  the  same 
as  when  there  is  an  RFP.  (CAM,  1995:  p.  D17) 
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When  production  is  necessary  on  items  not  previously  produced,  the  estimating 
department  (or  the  related  project  management  department)  solicits  a  preliminary  conceptual 
design  from  the  engineering  department.  The  preliminary  design  should  be  detailed  enough 
so  that  individual  parts  can  be  identified  and  numbered.  After  the  preliminary  design  has 
been  completed  and  reviewed,  a  work  breakdown  structure  (WBS)  is  prepared.  The  WBS 
is  a  matrix  that  organizes  and  describes  proposed  tasks  and  identifies  the  performing 
company  unit.  This  is  best  done  before  the  details  of  the  cost  estimate  request  are  finalized. 
(CAM,  1995:  p.  D 17) 

The  planning  process  includes  the  preparation  of  delivery  schedules,  staffing 
projections,  time-span  requirements,  and  funding  estimates.  Planning  is  a  cooperative  effort 
that  involves  the  estimating,  engineering,  and  production  control  departments.  (CAM,  1 995 : 
p.D17) 

"Grass-roots  estimates"  are  basic  estimates  of  labor,  material,  and  other  direct  costs 
developed  by  the  individual  business  units  that  will  actually  perform  the  work.  In  some 
cases,  departments  or  divisions  are  asked  to  generate  price  estimates.  When  this  occurs, 
special  care  must  be  used  to  ensure  that  sound  purchasing  considerations  such  as  competition 
and  quantity  discounts  are  applied  to  the  estimates.  (CAM,  1995:  p.  D17) 

The  engineering  department  usually  develops  labor  hour  estimates  for  any  potential 
items  to  be  produced  in-house.  These  estimates  are  normally  prepared  at  the  individual  part 
level.  The  manufacturing  department  uses  this  information  along  with  historical  data  to 
project  labor  requirements.  These  projections  may  be  broken  down  by  functional  area  and 
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cost  center.  A  variety  of  techniques  including  manpower  loading,  statistical  relationships, 
past  experience,  and  judgment  are  used  to  produce  the  estimates.  Additional  information 
such  as  program  schedules  and  configuration/performance  characteristics  from  preliminary 
and  final  engineering  design  drawings  may  be  worked  into  the  estimates.  In  all  cases,  the 
methods  used  to  produce  direct  labor  estimates  should  be  discernible,  and  supporting 
documentation  should  be  available  for  verification.  (CAM,  1995:  p.  D17) 

A  make-or-buy  committee  reviews  required  materials  and  associated  labor,  and 
determines  which  items  should  be  produced  internally.  In  some  instances,  decisions  will  be 
deferred  until  a  contract  award  is  made  and  further  design  effort  completed.  (CAM,  1995: 
p.D17) 

The  estimating  department  requests  purchasing  to  provide  estimates  for  all  potential 
buy  items.  The  purchasing  department  is  provided  with  the  best  available  specification  data 
from  engineering.  Delivery  requirements  are  provided  by  production  control.  Material  unit 
prices  (including  purchased  parts,  raw  material,  buy  to  drawing  items,  and  subcontract  items) 
are  obtained  by  the  purchasing  department  from  vendor  quotations,  current  purchase  orders, 
catalogs,  and  in  some  cases  statistical  methods.  Material  costs  are  usually  developed  by 
applying  these  prices  to  unit  quantities  in  a  bill  of  materials  provided  by  the  engineering  or 
manufacturing  department.  The  purchasing  and  estimating  departments  are  usually 
responsible  for  determining  appropriate  material  escalation  factors.  Escalation  is  either 
quoted  by  major  vendors  or  projected  using  specific  price  indices.  (CAM,  1995:  p.  D18) 
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Each  estimate  is  reviewed  and  approved  at  the  functional  level.  These  estimates  are 
then  submitted  to  the  estimating  department  that  assembles  the  complete  proposal. 
Estimating  personnel  integrate,  adjust,  and  analyze  estimates  for  accuracy  and  completeness. 
The  cost  estimate  is  summarized  further  by  functional  organization,  major  tasks,  and  other 
breakdowns  required  by  the  RFP.  When  all  direct  cost  elements  have  been  received  and 
properly  classified,  applicable  direct  labor  rates  and  indirect  expense  rates  are  applied  to 
complete  the  basic  cost  estimate.  These  rates  may  be  developed  by  either  the  estimating  or 
accounting  departments.  Fee  calculations  are  usually  applied  in  accordance  with  RFP 
guidance  and  company  pricing  policy.  (CAM,  1995:  p.  D18) 

Subsequent  to  initial  pricing  and  the  determination  of  profit  factors,  the  proposal  is 
reviewed  by  a  management  committee  usually  consisting  of  representatives  from  marketing, 
accounting,  plant  management,  estimating,  and  the  program  office.  The  committee 
scrutinizes  the  reasonableness  of  estimates,  overall  acceptability,  and  compatibility  with  the 
company's  business  strategy.  This  process  is  completed  by  the  formal  release  of  the  pricing 
proposal  and  supporting  rationale.  (CAM,  1995:  p.  D18) 

D.         PROBLEMS  IDENTIFIED  WITH  CONTRACTOR  ESTIMATING  SYSTEMS 

Numerous  studies  conducted  by  the  GAO  and  others  between  1 987  and  1 994  have 
focused  on  defective  pricing  and  inadequate  cost  estimating  systems.  Most  of  these  reports 
detail  that  problems  still  exist  in  this  area.  (GAO  93-8)  The  area  that  poses  the  most 
potential  risk  to  the  Government  is  subcontract  management.  The  majority  of  work  on  major 
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DoD  programs  is  increasingly  accomplished  by  subcontractors.  This  has  not  always  been 
the  case,  as  programs  in  the  1950's  subcontracted  out  as  little  as  nine  percent  of  total 
procurement  funds,  vice  the  50  to  75  percent  seen  today  (Mooney,  1991 :  p.  12).  Figure  1 
illustrates  the  growth  in  subcontracting  by  DoD  prime  contractors. 
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Figure  1.  Source:  Mooney,  1991 
The  first  situation  that  these  studies  have  identified  as  a  problem  area  in  cost 
estimates  is  that  prime  contractors  obtain  price  reductions  on  competitive  subcontracts  after 
negotiation  with  DoD.  The  DoD  requirement  for  a  contractor  to  obtain  cost  or  pricing  data 
supporting  subcontractor  proposals,  to  evaluate  those  data,  and  to  provide  the  evaluation 
results  to  the  Government  as  part  of  their  proposals  does  not  apply  to  subcontracts  awarded 
on  the  basis  of  adequate  price  competition  because  competition  is  presumed  to  produce  fair 
and  reasonable  prices.  (GAO  93-8, 1992:  p.  15)  This  can  be  seen  in  the  following  example: 
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GAO  reviewed  66  subcontracts  worth  $44  million  whose  estimates 
were  identified  by  a  contractor  as  having  been  competitively  obtained.  GAO 
found  that  after  "competitively"  soliciting  subcontract  prices  to  support  its 
proposals  to  the  Government,  the  contractor  had  resolicited  prices  and,  on  55 
of  the  subcontracts,  obtained  prices  that  were  $10.4  million  lower  than  what 
was  proposed  and  included  in  the  prime  contracts.  Conversely,  the  contractor 
awarded  10  subcontracts  at  prices  that  were  about  $1.5  million  more  than 
what  was  proposed  and  negotiated  in  the  prime  contracts.  Thus,  the  prime 
contractor's  actions  subsequent  to  negotiating  with  the  Government  resulted 
in  a  net  amount  of  $8.9  million  less  than  agreed  to  with  the  Government. 
(GAO/HR93-8,  1992:  p.  16) 


Prime  contractors  awarding  noncompetitive  subcontracts  at  a  cost  lower  than 

proposed  is  the  second  situation  involving  subcontracting  that  GAO  identified  as  a  problem. 

In  addition  to  negotiating  lower  prices  on  competitive  subcontracts,  some  prime  contractors 

negotiate  substantial  price  reductions  on  noncompetitive  subcontract  proposals  after 

completing  negotiations  with  the  Government  (GAO/HR  93-8,  1992:  p.  16). 

...  examined  12  noncompetitive  subcontract  estimates,  each  in  excess 
of  $1  million.  They  found  that  in  the  aggregate,  prime  contractors  had  made 
awards  on  these  estimates  for  about  $8.8  million  less  than  the  price 
negotiated  in  the  contracts  with  DoD.  (GAO/HR  93-8,  1992:  p.  17) 

DoD  must  have  an  effective  oversight  program  in  addition  to  requiring  sound 
contractor  cost  estimating  systems.  This  Governmental  oversight  program  should  do  two 
things.  First,  it  should  identify  and  audit  contractors  that  are  considered  high  risk  for 
overpricing  contracts.  Secondly,  this  program  should  ensure  timely  and  effective  corrective 
actions  when  deficiencies  are  found. 

DC AA  is  the  principal  agency  responsible  for  conducting  defective  pricing  audits  to 
determine  whether  contractors  have  complied  with  TINA.  Because  of  the  large  number  of 
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contracts  and  subcontracts  that  are  subject  to  audit,  DCAA  cannot  audit  all  contracts  and 

subcontractors  subject  to  TINA.  As  a  result,  DCAA  allocates  its  audit  resources  based  on 

its  assessment  of  risk.  In  assessing  risk,  DCAA  considers,  among  other  factors,  the  adequacy 

of  contractors'  cost  estimating  systems,  accounting  systems,  and  their  histories  of  defective 

pricing.  (GAO/HR  93-8,  1992:  p.  18) 

For  example,  high-risk,  fixed-price  contracts  worth  less  than  $10  million 
each,  DCAA's  selection  criterion  is  for  it  to  audit  1  in  15  contracts.  For 
medium-high  risk,  fixed-price  contracts,  its  selection  plan  calls  for  it  to  audit 
1  in  30  contracts  worth  less  than  $10  million  each.  (Richards,  1996:  p.  23) 

In  the  past,  DCAA  has  not  been  able  to  complete  all  the  audits  that  its  plans  call  for 

(GAO/HR  93-8,  1992:  p.  19).  The  majority  of  both  DCAA  auditors  and  contractors  agree 

that  the  level  of  auditing  pertaining  to  subcontract  management  oversight  has  decreased. 

Much  of  this  is  related  to  the  downsizing  of  DoD  and  a  resultant  shrinking  workforce. 

(Beaubien,  1995:  p.  1 14)  In  order  to  allocate  its  resources  to  contracts  with  the  highest  risk, 

DCAA  must  be  aware  of  the  universe  of  both  prime  and  subcontracts  subject  to  audit  under 

TINA  (GAO/HR  93-8,  1995:  p.  20).  This  is  not  always  accomplished. 

In  reviewing  a  sample  of  2 1 1  negotiated  subcontracts  having  a  dollar 
value  of  about  $337  million,  GAO  found  that  DCAA  was  not  aware  of  1 86, 
or  88  percent,  of  the  subcontracts.  The  186  subcontracts  were  about  $189 
million,  or  56  percent  of  the  total  value  of  subcontracts  in  our  sample. 
(GAO/HR  93-8,  1992:  p.  20) 

The  second  factor  for  a  quality  oversight  program  is  to  ensure  effective  and  timely 
corrective  action  once  deficiencies  are  noted.  This  can  only  be  done  with  an  effective 
follow-up  system. 


28 


We  reviewed  DoD's  audit  follow-up  system  and  found  that  the 
system  did  not  provide  accurate  and  complete  information  on  the  condition 
of  cost  estimating  systems  for  many  high  risk  contractors.  The  follow-up 
system  data  was  inadequate  in  three  areas.  The  system  (1)  was  missing 
reports  on  contractors;  (2)  understated  the  length  of  time  cost  estimating 
deficiencies  remained  uncorrected;  and  (3)  showed  contractors  had  corrected 
deficiencies,  when  our  review  showed  they  had  not.  (GAO/HR  93-8,  1992: 
p.  23) 


Timeliness  in  correcting  deficiencies  is  also  a  problem.  In  1 99 1 ,  GAO  reported  that 

many  cost  estimating  deficiencies  had  remained  uncorrected  for  long  periods  of  time  despite, 

the  1 988  revision  in  DoD's  regulation  requiring  major  contractors  to  establish  adequate  cost 

estimating   systems   and   DCAA   audits   reports   to    identify    estimating   deficiencies 

(GAO/NSIAD91-157, 1991:p.  7).  Toilluminatethispoint,in  1 994  GAO  auditors  reviewed 

the  status  of  30  contractors  identified  as  having  high  risk  cost  estimating  as  of  May  1992. 

As  of  January  1994,  the  30  contractors  had  corrected  or  potentially  corrected 
85  of  the  117  significant  deficiencies.  However,  the  32  remaining 
deficiencies  had  been  outstanding  an  average  of  3.8  years.  (GAO/NSIAD  94- 
153,  1994:  p.  3) 

The  results  of  not  having  an  effective  contractor  cost  estimating  and  Governmental  oversight 
program  costs  the  Government  additional  resources.  These  resources  are  both  monetary 
(higher  contract  prices)  and  time.  The  extra  steps  DoD  PCOs  take  are  frequently  time- 
consuming  and  costly. 

For  example,  in  December  1991,  DCAA  reported  that  a  certain 
contractor  had  five  significant  estimating  system  deficiencies.  DCAA  first 
reported  these  deficiencies  in  September  1990.  The  deficiencies  remained 
uncorrected  when,  in  June  1992,  the  contractor  submitted  a  $550  million 
proposal  to  DOD  that  was  based  on  costs  generated  by  the  estimating  system 
(GAO/NSIAD  94-153,  1994:  p.  5) 
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DCAA  and  Army  personnel  involved  in  the  negotiation  identified  numerous  errors  in  the 

proposal  that  would  have  significantly  increased  the  cost. 

DCAA's  audit  of  the  proposal  found  it  to  be  unacceptable  as  a  basis 
for  negotiating  a  fair  and  reasonable  price.  When  the  contractor  resubmitted 
data,  DCAA  found  much  of  it  still  unacceptable.  (GAO/NSIAD  94-153, 
1994:  p.  5) 

This  extra  probing  of  the  contractor's  proposal  was  time-consuming  because  the 

Army  was  not  sure  of  the  quality  of  the  data  submitted  and  needed  to  verify  nearly  all  the 

information  in  the  proposal.  As  a  result,  DCAA  used  more  audit  resources  than  it  normally 

would  have.  (GAO/NSIAD  94-153,  1994:  p.  5)    Army  contracting  officials  invested 

considerable  effort  in  obtaining  and  reviewing  information  from  the  contractor.  The  Army's 

Contracting   Officer  submitted  about   100  requests  to  the  contractor  for  additional 

information,  more  than  twice  the  normal  number  of  requests.  Army  contracting  officials  said 

that  "because  they  did  not  trust  the  contractor's  estimating  system,  reviewing  the  data  the 

contractor  provided  was  time-consuming"  (GAO/NSIAD  94-153,  1994:  p.  5). 

E.         SUMMARY 

Although  the  majority  of  actions  associated  with  contractor  estimating  systems  are 
conducted  by  the  ACO  and  DCAA  auditors,  the  PCO  is  significantly  impacted  by  these 
systems.  As  illustrated  above,  the  inadequacies  of  contractors'  estimating  systems  cost 
PCOs  in  funding  and  time.  Chapter  IV  will  present  the  survey  questionnaires  sent  to  PCOs, 
the  responses  received,  and  the  researcher's  analysis. 
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IV.  SURVEY  DATA  PRESENTATION  AND  ANALYSIS 

A.         INTRODUCTION 

This  segment  of  the  research  effort  is  centered  on  determining  the  specific  actions 
PCOs  take  when  notified  of  contractor  estimating  system  deficiencies  or  disapprovals.  The 
goal  was  to  obtain  factual  data  and  opinions  from  PCOs  regarding  how  they  received 
notification  and  what  actions  they  took  once  notified.  To  achieve  this  goal,  a  survey  was  sent 
to  PCOs  with  the  most  probable  exposure  to  deficient  contractor  estimating  systems. 

Surveys  were  mailed  in  early  September  to  272  PCOs  at  57  different  buying 
commands.  A  total  of  106  responses,  or  39%,  were  returned  representing  40  of  the  57 
commands.  These  40  commands  consisted  of  20  Air  Force,  ten  Navy,  eight  Army,  and  two 
DOD  agencies.  The  106  surveys  were  received  from  32  Air  Force  PCOs,  41  Navy  PCO's, 
3 1  Army  PCO's,  and  two  from  DoD  agency  PCOs.  Anonymity  for  the  individual  responder 
was  guaranteed  in  an  effort  to  ensure  that  PCOs  would  respond  to  the  questionnaires 
completely  and  honestly.  Therefore,  the  researcher  did  not  attempt  to  isolate  or  identify  any 
individual  responder  during  analysis  of  the  data  however,  individual  commands  were 
identified.  Appendix  A  provides  a  complete  copy  of  the  survey  questionnaire  utilized  for 
this  research  effort.  Appendix  B  provides  the  aggregate  responses  by  command. 

The  researcher  used  a  judgment  sample  to  conduct  the  survey.  A  judgment  sample 
targets  those  most  closely  related  to  the  issue  being  investigated.  Since  each  member  of  the 
population  does  not  have  an  equal  chance  of  being  chosen,  this  is  a  non-random  sampling 
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method.  The  reason  for  using  a  judgment  sample  was  to  survey  the  most  probable  PCOs 
with  exposure  to  estimating  system  deficiencies  and  disapprovals.  The  judgment  sample 
population  is  comprised  of  PCOs  for  DoD  acquisition  programs  with  RDT&E  budgets  of  $25 
million  or  greater.  The  researcher's  assumptions  about  the  sample  population  are:  (1)  the 
responder  will  have  a  high  experience  level,  and  (2)  the  value  of  contracts  the  responding 
PCOs  are  responsible  for  are  large  enough  to  warrant  review  of  contractors'  estimating 
systems. 

B.         SURVEY  COMPOSITION 

The  survey  questionnaire  consisted  of  nine  questions  that  were  designed  to  obtain 
both  objective  and  subjective  responses  concerning  contractor  cost  estimating  system 
deficiencies  and  disapprovals  with  respect  to  PCO  actions  once  notified.  Some  questions 
were  of  the  "open  ended"  type.  These  questions  were  included  to  gather  data  on  why  PCOs 
were  notified  and  what  action  they  took  once  notified.  Additionally,  several  demographic 
questions  were  posed  to  obtain  data  concerning  the  individual  respondent.  Demographic 
information  was  requested  to  provide  experience  level  and  contract  value  validity  of  the 
survey  results. 

The  survey  was  limited  to  one  page  and  the  researcher  estimated  that  it  would  require 
approximately  ten  minutes  to  complete  the  questionnaire.  The  researcher  felt  that  a  more 
concise  survey  would  increase  the  response  rate.  The  survey  asked  the  responder  to  answer 
questions  5,  6,  and  7  for  the  three  most  recent  occurrences  of  disapproved  or  deficiency 
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notification.  This  was  done  to  maximize  the  available  data  for  analysis.  The  researcher  felt 
that  PCOs  in  the  sample  would  have  had  multiple  experiences  with  deficient  estimating 
systems.  This  was  due  to  the  bias  of  the  sample  population  surveyed.  Allowing  them 
multiple  responses  to  these  questions  would  provide  more  information  for  the  purpose  of 
analysis. 

C.         SURVEY  RESPONSES  AND  ANALYSIS 

Each  question  from  the  survey  is  listed  below,  followed  by  a  summary  of  the  answers 
received  and  an  analysis. 

Question  1.  How  many  years  of  contracting  experience  do  you  have? 

a.  0-3  years 

b.  4-8  years 

c.  9-15  years 

d.  16-25  years 

e.  26  years  or  greater 
Responses:  The  survey  results  were: 

2%       0-3  years 

3%       4-8  years 

20%     9-16  years 

58%     16-25  years 

1 7%     26  years  or  greater 
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Of  the  1 06  participants  responding,  95%  had  nine  or  more  years  of  experience.  The 
most  frequent  response  from  all  Services  was  16-25  years  of  experience  as  shown  in  Figure 

2. 


Years  of  Experience 


0-3  years 
4-8years 
9-15  years 
16-25  years 
26  or  greater 


Source:  Developed  by  researcher 


Figure  2 

Analysis:  Question  1  was  asked  in  order  to  acquire  demographic  information 
regarding  the  sample  population.  It  is  intended  to  be  used  later  to  form  conclusions  about 
responses  to  other  survey  questions.  The  high  experience  level  of  the  participants  indicates 
that  these  PCOs  could  have  potentially  worked  on  several  proposals  produced  by  a 
contractor's  cost  estimating  system  throughout  their  career.  This  level  of  experience  should 
not  be  surprising.  PCOs  in  this  population  of  large  programs,  generally  have  many  years  of 
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experience  before  being  entrusted  with  this  degree  of  responsibility.  It  should  also  be  noted 
that  this  distribution  of  experience  does  not  necessarily  mirror  the  experience  levels  of  the 
entire  PCO  population. 

The  distribution  of  the  responses  from  each  Service  is  roughly  the  same  as  that 
depicted  in  Figure  2.  Five  respondents  had  eight  years  or  less  experience.  Two  respondents 
from  the  Air  Force,  two  from  the  Navy,  and  one  from  the  Army  fell  into  this  category.  All 
five  were  from  different  commands.  No  individual  command  displayed  characteristics  that 
would  distinguish  them  from  the  "norm." 

Question  2.  What  is  the  average  dollar  value  of  contracts  you  are  responsible  for  per 


year? 


a.  0-$l  million 

b.  $1-$  10  million 

c.  $10-$50  million 

d.  $50-$  100  million 

e.  $100  million  or  greater 
Responses:  Survey  results  were: 

7%       0-$l  million 
9%       $1-$  10  million 
19%     $10-$50  million 
17%     $50-$  100  million 
48%     $  1 00  million  or  greater 
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Eighty-four  percent  of  the  respondents  were  responsible  for  contracts  worth  $10 
million  or  more  per  year.  Figure  3  shows  the  most  frequent  response  to  the  question  was  in 
the  range  of  $  1 00  million  or  greater. 
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Figure  3 

Analysis:  Once  again  this  question  was  asked  to  gather  demographic  information 
about  the  sample  population.  It  is  intended  to  be  used  later  to  form  conclusions  about 
responses  to  other  survey  questions. 

Contractors'  estimating  systems  are  subject  to  Government  review  when  the 
preceding  year's  business  with  DoD  is  $50  million  or  more,  or  $10  million  or  more  when 
certified  cost  or  pricing  data  are  required.  (DFARS,  1996:  215.81 1-70)  Figure  3  shows  that 
the  majority  of  the  PCOs  surveyed  conduct  business  well  over  this  threshold.  This  indicates 
that  the  PCOs  sampled  will  have  a  high  probability  of  dealing  with  contractors  with 
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estimating  systems  that  are  subject  to  Government  review.  The  survey  results  support  the 
researcher's  second  assumption  about  the  sample  members;  that  is,  the  contract  value  is  large 
enough  to  be  subject  to  Government  oversight.  Again  it  should  be  noted  that  this  distribution 
does  not  necessarily  mirror  the  entire  PCO  population. 

The  distribution  of  the  responses  from  each  Service  is  roughly  the  same  as  the 
aggregate  depicted  in  Figure  3 .  Three  individual  commands  appear  to  comprise  an  abnormal 
share  of  the  more  than  $  1 00  million  category.  The  Army  Missile  Command  (MICOM),  the 
Naval  Air  Systems  Command  (NAVAIR),  and  the  Naval  Sea  Systems  Command 
(NAVSEA)  account  for  23  of  the  5 1  (45%)  responses  in  the  greater  than  $  1 00  million  range. 
The  other  55%  of  this  category  is  made  up  of  20  commands.  This  indicates  that,  for  the 
purposes  of  this  survey,  most  of  the  dollars  are  spent  in  the  three  aforementioned  commands. 

Question  3.  Have  you  ever  received  notification  of  a  contractor's  estimating  system 
having  deficiencies  or  being  disapproved? 

a.  Yes 

b.  No 
Responses:  Survey  results  are: 

69%)     Yes 
31%)     No 
Seventy-three  of  the  respondents  stated  they  had  received  notification  of  a  contractor 
having  deficiencies  or  disapproved  estimating  systems.  Figure  4  shows  these  results. 
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Analysis:  This  question  presents  a  fork  in  the  road  for  the  rest  of  the  research.  PCOs 
that  responded  with  a  "no"  answer  were  not  required  to  complete  any  further  questions  on 
the  survey.  Since  the  results  of  the  survey  question  are  based  on  a  judgment  sample,  readers 
should  not  conclude  that  69%  of  all  PCOs  are  dealing  with  contractors  who  have  deficient 
estimating  systems.  This  percentage  only  represents  the  sample  population.  The  validation 
of  the  judgment  sample  method  is  achieved  by  the  results  of  question  3.  Surveying  a 
population  whose  members  have  a  great  deal  of  experience  (nine  or  more  years)  and  are 
conducting  a  large  amount  of  business  ($10  million  or  more)  yielded  the  most  probable 
candidates  who  have  dealt  with  deficient  estimating  systems. 
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Figure  4. 

Using  a  weighted  average  from  the  data  gathered  in  questions  1  and  2,  PCOs  that  had 
not  been  notified  of  deficiencies  had  more  experience  and  were  responsible  for  higher  dollar 
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value  contracts.  Although  no  distinguishing  characteristics  were  noted  causing  this  group 
to  be  different  from  the  group  answering  "yes,"  the  industry  that  the  PCO  deals  in  may  be 
the  explanation  for  the  above  situation.  Among  the  PCOs  that  were  notified  of  deficiencies, 
the  years  of  experience  seems  to  be  the  more  dominant  variable  for  notification.  This 
situation  of  dominance  occurs  in  all  three  Services  and  across  a  variety  of  industries. 

The  responses  varied  seven  percent  between  the  Services.  The  Air  Force  had  the 
highest  percentage  of  "yes"  answers  (72%)  while  the  Army  had  the  least  (65%).  In  the 
opinion  of  the  researcher  this  variation  is  not  significant.  Three  commands  had  a  large 
impact  on  the  69%  overall  notification  rate.  MICOM,  NAVAIR,  and  NAVSEA  had  a  76% 
notification  rate  while  the  other  30  commands,  within  the  sample  of  "yes"  respondents, 
equated  to  64%. 

The  grouping  of  "no"  responses  had  no  distinguishing  characteristics.  They  were 
from  all  three  Services,  representing  1 8  commands.  These  commands  included  eight  Air 
Force,  five  Army,  four  Navy,  and  one  DoD  agency.  The  PCOs  in  this  group  had  varied 
experience  levels,  and  were  responsible  for  varied  dollar  contract  values. 

An  interesting  situation  surfaced  when  comparing  the  "yes"  and  "no"  responses  by 
command.  Ten  of  the  40  commands  that  sent  back  surveys  had  some  PCOs  responding 
"yes"  and  some  responding  "no".  Within  these  ten  commands,  43  PCOs  responded  "yes" 
and  21  responded  "no".  The  two  populations  of  "yes"  and  "no"  responders  are  responsible 
for  roughly  the  same  dollar  value  of  contracts.  This  situation  occurs  in  all  three  Services 
with  four  commands  from  the  Air  Force,  three  from  the  Navy,  and  three  from  the  Army. 
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This  would  not  be  surprising  if  the  commands  bought  totally  unrelated  products  with  a  wide 
supplier  base;  however,  this  may  not  be  the  case.  Many  of  the  markets  in  which  these  PCOs 
participate  have  few  suppliers.  This  information  by  itself  is  not  enough  to  draw  any 
conclusions. 

A  PCO  may  give  a  "no"  response  because  he  or  she  was  never  officially  notified  by 
the  ACO  or  DCAA  auditor  of  any  problems.  They  may  have  learned  from  other  PCOs  or 
may  have  found  out  through  other  methods  that  a  contractor  had  estimating  system  problems. 
If  this  was  the  case  and  they  took  actions  to  protect  the  Government  from  being  overcharged, 
this  survey  would  not  have  obtained  that  information.  This  should  be  considered  a  weakness 
of  the  survey  questionnaire. 

Question  4:  Why  were  you  notified? 

Responses:  This  was  an  open-ended  question  asked  to  determine  the  relationship 
between  the  PCO  and  the  Contractor.  The  returned  comments  were  varied  but  the  researcher 
grouped  them  into  four  general  categories.  These  categories  are:  ( 1 )  the  result  of  a  proposal 
evaluation  request,  (2)  the  PCO  is  on  a  standard  distribution  list  or  the  PCO  of  record  for  the 
program,  (3)  miscellaneous  answers,  and  (4)  no  response.  The  grouping  that  responders 
identified  most  often  (5 1  %)  was  the  proposal  evaluation  category.  Figure  5  details  how  the 
73  responses  are  distributed. 

Two  of  the  respondents  answered  the  question  in  a  way  that  did  not  place  them  in  any 
of  the  above  categories.  One  of  these  two  discussed  cost  estimating  relationships  of  a 
specific  contractor.  The  other  discussed  the  fact  that  the  contractor  they  were  dealing  with 
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did  not  have  an  estimating  system  at  all.  Neither  respondent  answered  the  question  of  why 
they  were  notified.  These  two  surveys  were  considered  "non-responsive  and  have  been 
included  in  the  "no  response  category." 

Analysis:  The  most  frequent  reason  a  PCO  was  notified  about  a  deficient  estimating 
system  was  an  ensuing  negotiation.  This  situation  is  caused  by  DoD's  policy  stating  that 
PCOs  shall  request  Field  Pricing  Reports  for: 


Why  PCO  was  Notified 
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Figure  5 

1 .  Fixed-Price  Proposals  Exceeding  $500,000. 

2.  Cost-type  proposals  exceeding  $500,000  from  offerors  with  significant 
estimating  deficiencies. 

3.  Cost -type  proposals  exceeding  $10  million  from  offerors  without 
significant  estimating  deficiencies.  (DFARS,  1996:  215.805-5) 

The  requirement  for  a  field  pricing  report  is  the  driver  for  the  high  response  in  this  category. 
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Informing  PCOs  of  a  contractor's  estimating  system  adequacy  on  a  proposal-by- 
proposal  basis  provides  the  most  accurate  assessment.  ACOs  are  to  semiannually  update  the 
status  of  the  contractor's  deficient  system  via  a  memorandum  to  their  DCMC  District  office. 
The  memorandum  may  also  be  enclosed  in  the  field  pricing  report.  PCOs  using  the  last 
complete  estimating  system  survey  as  a  basis  for  decisions  can  potentially  be  working  with 
two  to  three  year  old  information. 

The  second  most  frequent  response  was  a  result  of  being  on  a  standard  distribution 
list  or  being  the  PCO  of  record  for  the  program.  The  PCOs  in  this  grouping  are  either 
currently  in  performance  of  a  contract  with  the  contractor  or  have  conducted  a  significant 
amount  of  business  with  the  contractor.  PCOs  in  this  group  are  interested  in  the  contractor's 
estimating  system  for  potential  defective  pricing  and  contract  modification  purposes. 

The  third  grouping  of  responses  was  categorized  as  miscellaneous.  The  comments 
received  indicated  that  the  PCO  wanted  to  maintain  a  good  working  relationship  with  the 
ACO  and  that  generally  PCOs  are  notified  because  businesses  have  merged.  As  expected, 
the  low  number  of  responses  falling  into  the  miscellaneous  category  was  not  surprising. 
Notification  tends  to  come  from  the  usual  source  of  proposal  evaluations  or  from  being  the 
PCO  of  record  for  an  ongoing  contract. 

The  fourth  group  had  no  response.  It  included  the  two  surveys  listed  above  that  did 
not  give  a  logical  answer  for  the  question  and  one  left  unanswered. 

The  variation  among  responses  of  the  three  Services  was  not  significant.  Army  PCOs 
had  the  highest  percent  of  responses  in  the  proposal  evaluation  category  with  56%,  while 
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Navy  PCOs  had  the  lowest  at  46%.  None  of  the  commands  responding  had  any  outstanding 
characteristics. 

Question  5:  How  did  you  learn  of  the  deficiencies  or  disapprovals? 

a.  ACO 

b.  DCAA 

c.  Contractor 

d.  Cost  Analyst 

e.  Govt.  Tech/Eng 

f.  Other 
Response:  Survey  results  were: 

42%     ACO 

49%     DCAA 

5%       Contractor 

3%       Cost  Analyst 

0%       Govt.  Tech/Eng 

1%  other 
Questions  5, 6,  and  7  ask  for  information  about  the  three  most  recent  occurrences  of 
notification.  With  73  surveys  returned  answering  that  they  had  been  notified,  there  are  a  total 
of  219  responses  possible.  Of  the  219,  at  least  151  occurrences  were  reported  for  each  of 
these  questions.  The  majority  (125)  of  the  PCO's  responses  identified  only  one  source  of 
notification,  while  22  PCO's  responses  identified  two  sources  and  four  PCO's  responses 
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identified  three  or  more  sources  of  notification.  Figure  6  shows  that  the  vast  majority  of  the 
time  (91%)  the  PCOs  obtain  their  information  about  deficiencies  from  the  ACO  or  DCAA 
auditors.  The  other  nine  percent  of  the  time  they  acquired  it  from  the  contractor,  cost 
analysts,  or  other  sources. 
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Figure  6. 

Analysis:  The  survey  results  show  that  PCOs  were  informed  of  deficient  estimating 
systems  by  ACOs  42%  of  the  time.  The  Navy  and  Air  Force  identified  the  ACO  as  the  most 
frequent  source  with  49%  and  48%  respectively.    The  Army,  on  the  other  hand,  was 
informed  by  the  ACO  only  27%  of  the  time.  A  possible  reason  for  the  low  Army  percentage 
may  be  that  an  aircraft  carrier  costs  a  lot  more  and  takes  a  lot  longer  to  build  than  a  hand  held 
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radio.  Therefore,  a  contractor  building  an  aircraft  carrier  would  be  subject  to  more  review 
by  the  ACO  than  the  contractor  producing  the  hand  held  radio.  NAVSEA  and  NAVAIR 
accounted  for  80%  of  the  total  marking  of  this  response  for  the  Navy.  This  is  not  surprising 
considering  the  large  dollars  obligated  and  type  of  products  these  two  commands  buy.  No 
specific  Air  Force  command  was  a  driver  for  these  results. 

Notification  from  DCAA  was  the  most  frequent  response  overall.  It  accounted  for 
49%  of  all  answers.  Army  PCOs  identified  this  response  as  the  most  frequent  (65%).  This 
high  percentage  was  driven  by  one  command.  MICOM  accounted  for  21  of  the  Army's  34 
(62%)  responses.  These  21  responses  represented  12  Army  PCOs.  Eight  of  these  12 
identified  that  they  were  due  to  a  proposal  evaluation.  The  other  4  were  on  a  standard 
distribution  list  or  the  PCO  of  record.  This  indicates  that  MICOM  has  a  particularly  good 
communication  relationship  with  DCAA  and  that  the  length  of  a  procurement  has  an  affect 
on  who  notifies  the  PCO. 

The  fact  that  91%  of  the  PCOs  are  notified  by  the  ACO  or  DCAA  confirms  that  the 
PCOs  are  being  notified  by  the  proper  Government  personnel  as  established  in  the  FAR, 
DFARS,  and  DLA  Directives  discussed  in  Chapter  II.  The  fact  that  these  groups  are  doing 
the  majority  of  notification  does  not  speak  to  the  quality  of  the  information.  As  pointed  out 
in  GAO  audits,  many  ACOs  are  declaring  a  deficient  system  "adequate"  before  it  truly  is. 
(GAO/HR  93-8,  1992:  p.  23)  This  subject  of  ACO  adequacy  determination  is  an  area  in 
which  further  research  should  be  conducted. 
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Contractors  were  identified  as  the  third  most  frequent  source  of  notification,  at  the 
same  time  these  PCOs  also  identified  the  ACO.  The  dual  notification  was  identified  in  every 
case  that  the  contractor  was  listed  as  a  source  of  notification.  This  indicates  that  the  ACO 
notification  may  have  led  the  contractor  to  admit  to  the  problem.  Other  research  has 
identified  that  contractors  are  not  always  the  most  forthcoming  with  negative  information 
about  their  estimating  systems.  Based  on  a  survey  of  DC AA  auditors  and  contractors,  the 
auditors  believed  that  one  quarter  of  contractors  had  subcontract  estimating  deficiencies 
while  less  than  two  percent  of  the  contractors  thought  they  did.  (Beaubien,  1995:  p.  112) 

Cost  Analysts  were  identified  3%  of  the  time  in  the  survey  results.  Four  responses 
were  received  from  the  Air  Force  and  one  from  the  Navy.  Three  of  the  five  responses  were 
from  one  Air  Force  PCO.  The  intention  of  the  researcher  in  this  category  was  to  identify  cost 
analysts  within  the  PCO's  organization  or  program.  Nonetheless,  some  respondents  may 
have  answered  this  question  thinking  of  the  cost  analysts  within  the  DCMC  structure. 
Regardless  of  any  misinterpretation  of  the  meaning  of  the  answers  provided,  the  percent  of 
the  total  is  not  material. 

None  of  the  PCOs  identified  the  Government  Technical  and  Engineering  community 
as  a  source  of  information  about  a  contractor's  estimating  system.  Although  the  researcher 
included  them  as  an  option,  the  fact  that  no  PCO  identified  them  as  a  source  is  not  surprising. 
In  the  researcher's  opinion,  this  shows  that  the  Technical  and  Engineering  community  is 
more  interested  in  what  is  being  proposed  than  how  much  the  proposal  will  cost. 
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The  "other"  option  was  provided  to  try  to  eliminate  any  bias  that  would  be  created 
by  providing  specific  answers  to  the  question.  The  researcher's  hypothesis  was  that  some 
PCOs  would  have  learned  of  contractors'  estimating  system  problems  from  other  PCOs  or 
through  their  contracting  chain  of  command.  This  did  not  turn  out  to  be  the  case.  One 
possible  reason  for  the  low  response  in  this  category  is  that  PCOs  may  feel  self-conscious 
about  having  to  admit  that  their  chain  of  command  had  to  notify  them  of  deficiencies 
regarding  their  contractor. 

The  "other"  category  received  two  responses.  The  first  was  that  a  PCO  had 
discovered  the  deficiency  himself  and  informed  the  ACO.  This  instance  has  to  be  one  that 
slipped  through  the  process,  however,  the  second  instance  was  identified  by  the  Army  Audit 
Agency.  Based  on  the  other  answers  by  this  respondent  throughout  the  rest  of  the  survey, 
this  was  not  the  first  indication  of  a  problem. 

When  examining  the  responses  to  this  question,  the  researcher  noticed  that  some 
questionnaires  identified  multiple  sources  of  notification.  This  examination  revealed  that 
125  of  the  total  151  (83%)  identified  a  single  source  of  the  notification.  Twenty-six  of  the 
total  151  (17%)  identified  multiple  sources  of  notification.  This  situation  shows  a  weakness 
in  the  researcher's  questionnaire.  The  question  should  have  asked  the  responder  to  only 
answer  one  per  notification  occurrence.  The  researcher's  intent  was  to  identify  from  whom 
the  PCO  first  learned  of  the  deficient  estimating  system. 

Question  6.  What  actions  did  you  take  once  notified? 

Responses:  Survey  results  were: 


47 


62%     Internal  to  Government 


1 7%     Direct  with  Contractor 


16%     Both 


5%       Miscellaneous 
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Figure  7. 

This  was  an  open-ended  question  asked  to  determine  the  actions  taken  by  the  PCO. 
The  returned  comments  were  varied  but  the  researcher  grouped  them  into  four  general 
categories  identified  in  Figure  7.  The  most  frequent  response  fell  into  the  grouping  classified 
as  Government  internal  action.  Responses  in  this  category  include  the  PCO  determining  the 
impact  of  the  deficiency,  the  PCO  discussing  the  deficiency  with  the  ACO  or  DCAA,  the 
PCO  informing  their  chain  of  command,  and  others  that  indicated  the  PCO  actions  were 
confined  to  dealings  within  the  Government.  The  second  most  common  response  grouping 
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was  direct  communications  with  the  contractor.  This  category  consisted  of  responses  that 
indicated  the  PCO  contacted  the  contractor  directly  and  discussed  or  resolved  the  issues 
without  other  identified  Government  intervention.  The  third  grouping,  labeled  as  both, 
consists  of  responses  indicating  the  PCO  formed  some  kind  of  working  or  discussion  group. 
The  group  would  consist  of,  not  only  the  PCO  and  contractor,  but  also  the  ACO  and  DC  AA 
auditors.  The  last  grouping  is  labeled  as  miscellaneous  and  consists  of  six  responses  that  did 
not  fit  into  the  other  categories. 

Analysis:  This  question  was  included  to  help  answer  the  primary  research  question 
of  what  actions  PCOs  take  when  notified  of  deficient  contractor  estimating  systems.  The 
researcher  observed  from  the  survey  that  95%  of  the  time  the  PCO  took  some  form  of 
positive  action.  Therefore  it  can  be  said  that  receiving  notification  of  a  deficiency  in  a 
contractor's  estimating  system  has  an  impact  on  the  PCO  and  the  majority  of  the  time  his  or 
her  response  is  internal  to  the  Government. 

A  total  of  1 68  responses  were  received  for  this  question.  PCOs  took  actions  internal 
to  the  Government  101  times  or  60%  of  the  time.  Twenty-  six  responses  (16%)  stated  that 
the  PCOs  worked  directly  with  the  contractor.  Twenty  percent  of  the  time  the  PCOs  worked 
as  a  team  with  the  ACO,  the  DCAA,  and  the  contractor.  There  was  no  significant  variation 
of  results  between  the  Services  for  any  of  these  categories. 

Question  6  asked  all  respondents  to  identify  their  actions  regardless  of  whether  they 
were  in  negotiations  or  not.  Question  7  asks  of  the  PCO's  actions  if  in  negotiations. 
Therefore,  PCOs  answering  question  6  that  are  in  negotiations  would  also  answer  question 
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7.  To  unmask  the  affect  of  potential  double  counting,  the  researcher  eliminated  all  PCOs  in 
a  negotiation  status  from  question  6.  This  new  grouping  consists  of  only  PCOs  on  a  standard 
distribution  list  or  PCOs  of  record  for  programs,  and  miscellaneous  and  no  responses  from 
question  4.  The  actions  these  PCOs  took  can  be  grouped  into  the  same  categories  identified 
in  question  4.  Fifty-five  percent  of  them  took  actions  internal  to  the  Government.  Eleven 
percent  worked  directly  with  the  contractor  to  resolve  the  issues  and  26%  worked  both  with 
the  contractor  and  other  Government  personnel  such  as  the  ACO  and  DCAA.  This 
distribution  is  roughly  the  same  for  all  PCOs  in  the  sample  population.  There  is  no  evidence 
to  indicate  why  this  situation  occurs.  There  was  no  significant  variation  between  the 
Services  in  this  grouping. 

Two  of  the  responses  in  the  miscellaneous  category  stood  out  from  the  others.  The 
PCOs  in  these  cases  stated  that  they  awarded  the  contract  to  the  next  higher  priced 
contractor.  From  a  contractor's  perspective,  this  action  would  grab  one's  attention  quickly. 
The  viability  of  the  company  may  rest  with  Government  contracts.  Businesses  that  can  not 
compete  because  of  their  estimating  systems  have  significant  problems. 

Question  7.  If  in  negotiations,  how  did  you  proceed? 

a.  No  change  in  plan. 

b.  Allowed  the  contractor  additional  time  to  correct  the  deficiency  and 
submit  a  corrected  proposal. 

c.  Considered  another  type  of  contract. 
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d.  Used  additional  cost  analysis  to  determine  the  reasonableness  of  the 
cost  elements  affected  by  the  system's  deficiency. 

e.  Segregated  the  questionable  areas  as  a  cost  reimbursable  line  item. 

f.  Reduced  the  negotiation  objective  for  profit  or  fee. 

g.  Included  a  contract  reopener  clause  that  provides  for  adjustment  of  the 
contract  amount  after  award. 

h.  Other. 
Responses:  Survey  results  were: 
1 3%     No  change  in  plan. 
25%     Allowed  the  contractor  additional  time  to  correct  the  deficiency  and 

submit  a  corrected  proposal. 
2%       Considered  another  type  of  contract. 
3 1  %     Used  additional  cost  analysis  to  determine  the  reasonableness  of  the 

cost  elements  affected  by  the  system's  deficiency. 
4%       Segregated  the  questionable  areas  as  a  cost  reimbursable  line  item. 
7%       Reduced  the  negotiation  objective  for  profit  or  fee. 
12%     Included  a  contract  reopener  clause  that  provides  for  adjustment  of 

the  contract  amount  after  award. 
6%       Other. 
Figure  8  shows  that  the  majority  of  respondents  stated  they  used  additional  cost 
analysis.    The  second  most  frequent  response  was  allowing  additional  time  to  correct  the 
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deficiency  then  resubmit  a  corrected  proposal.    Seventy-four  percent  of  the  responses 
identified  only  one  action  taken  while  26%  used  two  or  more  actions  in  tandem. 
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Figure  8. 
AnalysisrThis  question  was  intended  only  for  PCOs  in  the  negotiations  phase  of  a 
procurement.  The  responses  to  this  question  should  help  answer  one  of  the  subsidiary 
research  questions  of  determining  whether  the  alternatives  provided  in  the  DFARS  motivate 
contractors  so  that  the  Government  obtains  a  fair  and  reasonable  contract. 

The  use  of  additional  cost  analysis  to  determine  the  reasonableness  of  the  cost 
element  affected  by  the  system's  deficiency  was  the  most  frequent  response  overall.  Army 
and  Navy  PCOs  used  this  option  35%  and  29%  of  the  time  respectively.  The  Air  Force 
ranked  it  as  their  second  most  frequent  option.  One  Navy  PCO  provided  a  comment  that  this 
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option  was  difficult  for  some  PCOs.  This  may  be  the  case  if  time  is  of  the  essence.  PCOs 
are  dealing  with  a  variety  of  timing  issues  on  a  regular  basis  that  may  preclude  the  use  of  this 
option.  Expiring  appropriations,political  influences  in  the  procurement  process,  and  urgency 
of  need  are  a  few  of  the  issues  that  may  make  this  option  unattractive. 

PCOs  responding  to  the  survey  seem  to  use  this  option  in  tandem  with  others.  Thirty- 
three  percent  of  the  time,  when  this  option  was  chosen,  another  option  was  chosen  with  it. 
Every  option  offered  in  the  survey  was  identified  as  an  action  used  in  tandem  with  additional 
cost  analysis.  The  most  frequent  action  used  in  concert  with  additional  cost  analysis  was 
allowing  the  contractor  additional  time  to  correct  the  deficiency  and  resubmit  a  corrected 
proposal. 

Allowing  the  contractor  additional  time  to  correct  the  deficiency  and  submit  a 
corrected  proposal  was  the  second  most  frequent  response.  The  Army  and  the  Navy  used 
this  option  25%  and  21%  respectively  while  the  Air  Force  ranked  it  as  their  most  frequent 
response  with  33%.  No  particular  command  statistically  stood  out  from  the  others.  Analysis 
shows  that  of  the  46  responses  that  used  this  option,  17  (37%)  used  it  in  combination  with 
another  action.  This  trend  was  seen  equally  among  the  three  Services.  When  this  option  was 
used  in  combination  with  another,  it  was  most  frequently  with  the  option  of  additional  cost 
analysis. 

The  option  of  "no  change  in  plan"  was  included  in  the  answer  set  for  PCOs  that  were 
either  dealing  with  estimating  deficiencies  that  were  minor  and  have  no  material  affect  on 
negotiations,  or  were  dealing  with  a  contractor  who  was  a  sole  source.  In  these  cases  the 
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PCOs  had  no  alternative  but  to  continue  with  the  process.  The  responses  stating  "no 
change"  noted  that  either  the  deficiency  was  minor  or  the  ACO  was  already  working  with 
the  contractor  to  correct  the  system.  One  Navy  PCO  did  note  that  he  or  she  would  only  use 
this  action  when  negotiating  a  cost  type  contract  and  the  potential  impact  of  fee  is  minimal. 

Including  a  contract  reopener  clause  that  provides  for  adjustment  of  the  contract 
amount  after  award  received  22  responses.  Eight  Navy,  seven  Army,  four  Air  Force,  and  one 
DoD  agency  PCO  used  this  option  for  the  total  of  22.  The  Navy  and  Army  accounted  for  1 8 
(82%)  of  the  responses.  Within  the  Navy,  NAVSEA  PCOs  used  this  option  six  times.  This 
appears  to  be  a  standard  policy  for  NAVSEA  PCOs.  One  NAVAIR  PCO  commented  that 
this  option  has  been  used  for  years  at  NAVAIR;  yet  this  was  the  only  response  from 
NAVAIR  that  identified  this  option.  MICOM  PCOs  used  this  option  six  times.  With  one- 
third  of  the  MICOM  PCOs  using  this  option  it  appears  that  this  is  somewhat  of  a  standard 
procedure  for  this  command. 

The  researcher  observed  that  PCOs  use  actions  that  would  have  the  contractor  fix  the 
problem  before  an  award  is  made  rather  than  after.  This  can  be  seen  in  the  fact  that  the  use 
of  additional  time  and  additional  cost  analysis  is  used  far  more  frequently  than  a  reopener 
clause.  A  second  reason  for  low  overall  response  to  this  option  is  the  fact  that  the  DFARS 
requires  that  PCOs  including  a  reopener  clause  are  responsible  for  negotiating  price 
adjustments  required  by  the  clause.  This  requires  the  PCO  to  be  more  involved  in  the 
administrative  functions  of  the  contract  than  normal.  PCOs  prefer  to  award  contracts  after 
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contractors  have  fixed  estimating  problems  rather  than  awarding  and  providing  additional 
oversight  for  the  deficient  area. 

Reducing  the  negotiation  objective  for  profit  or  fee  received  13  (seven  percent) 
responses.  Six  Navy,  two  Air  Force,  and  one  Army  PCO  used  this  option.  All  nine  PCOs 
who  responded  used  this  option  in  concert  with  another  option.  There  was  no  correlation 
between  the  commands  using  this  option  or  the  other  option  used.  This  option  was  used  as 
part  of  the  weighted  guidelines  under  the  cost  control  contractor  risk  factor.  The  evaluation 
criteria  for  below  normal  conditions  are: 

1.  The  contractor's  cost  estimating  system  is  marginal. 

2.  The  contractor  has  made  minimal  effort  to  initiate  cost  reduction  programs. 

3.  The  contractor's  cost  proposal  is  inadequate. 

4.  The  contractor  has  a  record  of  cost  overruns  or  other  indications  of  unreliable 
cost  estimating  and  lack  of  cost  control. 

PCOs  using  this  option  directly  affect  the  companies'  profits.  This  is  an  option  that  assumes 
the  contract  will  be  awarded  before  the  contractor  fixes  the  estimating  system  problem.  As 
seen  by  the  relative  low  response,  this  confirms  that  PCOs  prefer  to  have  the  estimating 
system  fixed  prior  to  award. 

Segregating  the  questionable  areas  as  a  cost-reimbursable  line  item  has  the  second 
lowest  response  rate.  It  was  used  six  times  and  four  of  them  were  as  a  combination  of 
options.  This  option  would  be  a  good  alternative  if  the  contractor  has  an  accounting  system 
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deficiency.  The  cost-reimbursable  feature  allows  the  PCO  visibility  of  all  costs  within  that 
element  thus  keeping  tabs  on  the  contractor. 

Considering  another  contract  type  was  the  least  used  option.  Three  Air  Force,  one 
Navy,  and  one  Army  PCO  identified  this  as  a  viable  option.  The  low  usage  of  this  option 
could  be  due  to  the  fact  that  other  factors  were  more  weighty  in  the  selection  of  a  contract 
type  than  the  risk  of  inaccurately  priced  proposals.  An  example  of  this  may  be  a  research 
and  development  contract  that  has  substantial  development  risk.  Changing  from  a  cost  type 
contract  to  a  firm  fixed-priced  type  may  be  impracticable. 

Ten  responses  (five  percent  of  the  total)  used  other  actions.  This  group  was  made  up 
of  five  Navy,  three  Air  Force,  and  two  Army  PCOs.  They  included  a  variety  of  comments 
and  ideas  such  as: 

1 .  The  PCO  stopped  all  activity  with  the  contractor  until  the  system  was 
revalidated  (three  responses). 

2.  Awarded  the  contract  to  the  next  competitor  at  a  higher  price  (two  responses). 

3.  Documented  the  Business  Clearance  Memorandum  (BCM)  so  the 
Government's  rights  were  protected  in  the  event  the  deficiency  results  in  a 

defective  pricing  action. 

4.  Utilized  a  Department  of  Justice  settlement  of  a  fraud  case  to  extract  a 
commitment  from  the  contractor  to  fix  his  deficiencies. 

5.  Allowed  ACO  to  include  in  a  "global  settlement"  of  issue  (all  active  contracts 
and  proposed  effort). 
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6.  Contractor  withdrew  proposal  (two  responses). 

Thirteen  actions  were  taken  by  PCOs  to  reduce  the  profit  objective  and  seven 
(included  in  the  other  categories)  actions  led  to  not  awarding  a  contract  to  that  company. 
These  20  (11%)  actions  taken  by  the  PCO  directly  impacted  the  company's  profit  and 
business  base.  This  high  percentage  of  PCOs  who  directly  affect  the  businesses'  bottom  line 
surprised  the  researcher.  This  severe  action  indicates  that  the  contractors  involved  must  have 
significant  deficiencies. 

Examining  the  survey  data  as  a  whole  shows  that  using  additional  cost  analysis  and 
allowing  the  contractor  additional  time  to  correct  the  deficiency  then  resubmit  a  corrected 
proposal  were  the  most  frequent  responses  by  all  Services.  Together  they  account  for  56% 
of  the  total  responses  received.  The  dimension  of  time  plays  a  critical  role  in  looking  at  these 
responses.  Allowing  the  contractor  additional  time  to  correct  the  deficiency  then  resubmit 
a  corrected  proposal  and  using  additional  cost  analysis  adds  to  the  procurement  process  time. 
Lengthening  this  process  is  counter  to  the  current  emphasis  within  DoD  to  reduce  the  time 
it  takes  to  buy  and  field  products. 

Question  8  &  9.  Did  the  actions  you  took  motivate  the  contractor  to  give  the 
Government  a  fair  and  reasonable  price?  Why? 

a.  Yes 

b.  No 

Response:  Survey  results  are: 
77%     Yes 
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23%     No 
A  total  of  43  respondents  stated  their  actions  did  motivate  the  contractor  and  the 
Government  did  receive  a  fair  and  reasonable  contract  price.  Fourteen  respondents  gave  a 
negative  answer.  Additionally,  16  responses  were  received  with  this  question  unanswered. 
Figure  9  displays  these  results. 
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Source:  Developed  by  researcher 


Figure  9. 
Analysis:  These  two  questions  were  included  in  the  questionnaire  to  evaluate  if 
PCOs  felt  the  actions  they  took  in  question  7  resulted  in  the  Government  receiving  a  fair  and 
reasonable  contract  price  and  why.  By  analyzing  the  two  questions  together,  six 
combinations  of  answers  are  possible.  The  researcher  will  analyze  each  combination 
individually.    The  first  grouping  of  PCOs  are  ones  that  answered  "yes"  in  question  8  and 
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gave  comments  in  question  9.  This  group  contained  35  responses  that  consisted  of  1 8  Army, 
1 1  Navy,  and  six  Air  Force  PCOs.  Fifteen  of  these  responses  can  be  loosely  grouped  around 
the  idea  that  the  PCO  believed  the  contractors  were  concerned  about  the  loss  of  current  or 
future  business  if  the  system's  problems  were  not  fixed.  Most  of  these  comments  stated  that 
the  PCOs  were  not  going  to  award  a  contract  until  the  deficiency  was  fixed  or  the 
disapproval  was  removed.  These  comments  support  the  high  usage  of  additional  time  to 
correct  and  additional  cost  analysis  and  low  usage  of  reopener  clauses,  segregating  the 
questionable  area,  and  reducing  profit  or  fee  identified  in  question  7.  This  information 
indicates  that  the  PCO  is  operating  in  an  environment  where  time  is  not  a  critical  element  and 
the  PCO  can  afford  the  time  to  allow  the  contractor  to  fix  the  estimating  system. 

The  second  grouping  consists  of  PCOs  stating  "yes"  to  question  8  but  did  not  respond 
to  question  9.  This  group  was  made  up  of  four  Army,  two  Navy,  one  Air  Force,  and  one 
DoD  agency  PCO.  Since  no  comments  were  provided  by  these  respondersthe  researcher  can 
only  assume  that  their  comments  would  be  similar  to  the  previous  group. 

The  third  group  included  PCOs  that  were  in  the  negotiation  phase  but  were  unsure 
if  their  actions  motivated  the  contractor.  This  group  consisted  of  six  Navy,  four  Army,  and 
two  Air  Force  PCOs.  The  majority  of  these  responders  stated  that:  (1)  the  deficiency  was 
minor  in  impact,  (2)  they  used  additional  cost  analysis  in  question  7,  or  (3)  they  allowed  the 
contractor  additional  time  to  correct  the  problem  and  resubmit  the  proposal.  The  researcher 
observed  that  PCOs  taking  a  more  neutral  action  such  as  two  and  three  above  were  more 
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uncertain  as  to  the  affect  their  actions  had  on  the  contractor  than  PCOs  that  took  a  stronger 
stance  as  identified  in  the  first  group. 

The  fourth  group  did  not  answer  either  question  eight  or  nine  because  they  were  not 
in  negotiations.  This  collection  of  answers  consisted  of  two  Air  Force,  one  Army,  and  one 
Navy  PCOs. 

The  fifth  group  stated  their  actions  did  not  motivate  the  contractor  and  did  not 
provide  a  reason  why.  One  response  fell  into  this  category.  It  was  from  an  Army  PCO  who 
stated  in  question  7  that  the  deficiency  did  not  have  a  material  affect.  The  responder  used 
the  action  "no  change  in  plan"  as  the  action  taken  in  question  7.  Since  no  action  was  taken, 
then  the  contractor  was  not  motivated. 

The  last  collection  of  answers  consisted  of  PCOs  that  did  not  feel  their  actions 
motivated  the  contractor  and  provided  the  reasons  why  they  felt  that  way.  This  group 
included  eight  Navy,  three  Army,  and  two  Air  Force  PCOs.  The  majority  (11)  from  this 
group  stated  that  the  deficiency  was  not  a  factor  in  pricing  or  the  contractor  did  not  agree 
with  DCAA.  The  two  remaining  PCOs  were  dealing  with  sole  source  contractors.  These 
PCOs  used  more  neutral  actions  of  additional  cost  analysis  and  allowing  the  contractor 
additional  time  to  correct  the  problem  then  resubmit  the  proposal.  The  researcher  observed 
that  when  a  PCO  is  dealing  with  a  sole  source  they  are  more  likely  to  use  less  aggressive 
actions  toward  the  contractor  than  if  operating  in  a  multi-competitor  market. 
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D.  SUMMARY 

This  chapter  presented  and  analyzed  the  data  obtained  from  the  survey  sent  to  various 
PCOs  of  large  programs  for  the  three  Services  and  selected  DoD  agencies.  This  sample 
population  was  chosen  because  the  researcher  felt  the  members  had  the  highest  probability 
of  dealing  with  contractor  estimating  systems.  Overall,  the  comments  received  appeared  to 
be  open  and  informative.  The  level  of  responses  was  higher  than  the  researcher  anticipated. 

Analyzing  the  returned  comments  caused  the  researcher  to  rethink  his  goal  of 
maintaining  only  a  one  page  questionnaire.  More  in-depth  information  could  have  been 
obtained  by  including  a  few  more  questions  despite  any  decline  in  the  return  rate.  After 
evaluating  the  returned  responses,  the  researcher  found  some  weaknesses  with  the 
questionnaire.  First,  it  did  not  give  PCOs  that  had  not  ever  been  "officially"  notified  of  a 
deficient  contractor  estimating  system  the  ability  to  provide  information  on  actions  they  may 
have  taken  to  protect  the  Government  anyway.  Secondly,  the  questionnaire  did  not  ask  the 
PCO  to  provide  their  rationale  on  why  they  took  the  actions  they  did.  Lastly,  the 
questionnaire  failed  to  direct  the  respondents  to  provide  only  one  answer  for  each  occurrence 
in  question  5. 

Chapter  V  will  provide  conclusions,  recommendations,  answers  to  the  research 
questions,  and  recommendations  for  further  research. 
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V.  CONCLUSIONS  AND  RECOMMENDATIONS 

A.         INTRODUCTION 

Reliable  cost  estimating  systems  are  a  key  safeguard  for  the  Government  to  obtain 
fair  and  reasonable  contract  prices.  When  a  contract  price  is  higher  because  it  is  based  on 
inaccurate,  incomplete,  and  non-current  data  the  Government  has  the  right  to  a  price 
reduction  under  the  provisions  of  TINA.  When  contracts  are  overpriced  because  of  poor 
estimating  the  Government  may  not  recover  the  overstated  price  under  TINA.  Numerous 
audits  conducted  by  GAO  over  the  last  several  years  indicate  there  are  significant  problems 
and  weaknesses  in  many  contractors'  cost  estimating  systems. 

The  objectives  of  this  research  effort  were:  (1)  to  determine  how  PCOs  obtain 
informatio  n  of  a  contractor  poor  cost  estimating  system  and  (2)  to  determine  how  this 
information  is  used  by  the  PCO  once  obtained.  For  the  Government  to  receive  a  fair  and 
reasonable  contract  price  the  PCO  must  know  if  the  contractor's  proposal  was  generated  by 
an  adequate  estimating  system.  If  the  PCO  is  unaware  of  the  adequacy  of  the  system  the 
Government  cannot  recover  any  overstated  amounts  from  the  contractor  once  the  negotiation 
is  completed.  The  information  about  the  adequacy  of  a  contractor's  estimating  system  is  a 
key  defense  that  PCOs  must  use  to  ensure  taxpayers  funds  are  spent  wisely.  The  researcher 
conducted  a  survey  of  PCOs  in  high  dollar  value  DoD  programs  and  projects  to  achieve  the 
stated  objectives.    These  PCOs  were  sent  a  nine-question  survey    estimated  to  take  ten 
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minutes  to  complete.    The  researcher  felt  that  a  brief  questionnaire  would  increase  the 
response  rate. 

B.         CONCLUSIONS 

This  section  discusses  conclusions  drawn  as  a  result  of  this  research  effort. 

1.  PCOs  are  informed  of  deficient  contractor  estimating  systems  by  the  ACO 
and  DCAA  auditor. 

The  procedures  in  the  DFARS  and  DLA  directives  place  the  responsibility  of 
estimating  system  adequacy  with  the  ACO.  The  DCAA  auditors  are  to  assist  the  ACO  with 
this  function  by  conducting  estimating  system  surveys  once  every  three  years.  These 
procedures  were  discussed  in  earlier  chapters  of  this  thesis.  The  results  of  the  researcher's 
survey  indicate  that  the  ACO  and  DCAA  auditor  inform  PCOs  the  vast  majority  of  the  time. 
For  the  research  sample,  this  accounted  for  91%.  This  confirms  that  the  DFARS  and  DLA 
procedures  discussed  in  Chapter  II  are  being  followed. 

2.  PCOs  are  typically  notified  of  poor  estimating  systems  during  proposal 
evaluation  and  negotiation  situations. 

The  DFARS  requires  PCOs  to  request  field  pricing  reports  for  procurement  actions 
over  prescribed  thresholds.  Many  of  the  PCOs  stated  they  had  been  informed  of  the 
adequacy  of  the  contractor' s  estimating  system  via  these  reports.  Additionally,  some  ACOs 
include  their  semi-annual  ACO  memorandum  updating  the  status  of  the  contractor's 
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corrective  action  plan  in  the  report.  For  the  sample  population,  this  method  was  used  5 1  % 
of  the  time. 

3.  PCOs  take  actions  internal  to  the  Government  when  notified  of  estimating 
systems  problems. 

PCOs  typically  take  actions  internal  to  the  Government  when  notified  of  contractors' 
deficient  estimating  systems.  These  actions  include  determining  the  impact  of  the  deficiency 
or  disapproval,  holding  discussions  with  the  ACO  and  DCAA  auditor,  and  informing  the 
PCO's  chain  of  command.  The  impact  of  the  deficient  system  is  based  on  the  contractual 
relationship  between  the  PCO  and  the  contractor.  Discussions  with  the  ACO  and  DCAA  are 
generally  conducted  to  determine  the  status  of  the  corrective  actions  performed  by  the 
contractor.  The  PCOs  informed  their  chain  of  command  in  an  attempt  to  formulate  a 
coordinated  response  from  a  command  perspective.  The  survey  conducted  for  this  research 
determined  that  62%  of  the  actions  taken  by  the  survey  participates  were  internal  to  the 
Government. 

4.  Generally  PCOs  use  either  additional  cost  analysis  or  additional  time  to 
correct  systems  deficiencies  when  in  negotiations. 

The  DFARS  provides  various  alternative  action  for  PCO  to  use  when  dealing  with 
a  contractor  that  has  a  deficient  estimating  system.  PCOs  in  the  negotiations  phase  of  a 
procurement,  utilize  additional  cost  analysis  on  proposals  generated  from  these  deficient 
estimating  systems.  Secondly,  they  allow  the  contractor  additional  time  to  correct  the 
deficiency  and  submit  a  corrected  proposal.  Fifty-six  percent  of  the  sample  participants  used 
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one  of  these  two  methods.  Both  actions  may  lengthen  the  negotiation  phase  of  a 
procurement.  Utilizing  additional  cost  analysis  requires  additional  resources  to  be  used  when 
they  could  be  focused  in  other  areas. 

5.  PCOs  believe  their  actions  motivated  the  contractor  so  that  the  Government 
received  a  fair  and  reasonable  price. 

PCOs  use  a  wide  range  of  actions  to  ensure  contractors  are  sensitive  to  the 
Government's  concerns.  One  of  these  concerns  is  fair  and  reasonable  contract  prices.  When 
the  contract  price  is  negotiated  based  on  a  deficient  estimate,  the  PCO  must  take  actions  to 
protect  the  Government.  The  majority  (77%)  of  PCOs  in  the  sample  believe  that  the  actions 
they  took  motivated  the  contractor  so  the  Government  obtained  a  fair  and  reasonable  price. 
Although  the  rationale  given  varied  greatly,  many  PCOs  felt  contractors  were  concerned 
about  losing  current  or  future  business  if  corrections  to  their  estimating  systems  were  not 
made. 

6.  PCOs  do  not  receive  notification  about  contractor  estimating  system 
problems  from  their  contracting  chain  of  command. 

This  conclusion  is  based  on  three  pieces  of  independent  data  provided  in  the  returned 
surveys.  First,  none  of  the  PCOs  listed  their  contracting  chain  of  command  (buying 
command  management  or  contracting  directorate)  as  a  source  of  information  about  a 
contractor's  estimating  system  in  question  5.  Secondly,  four  separate  PCOs  stated  that  they 
had  informed  their  upper  management  so  that  a  coordinated  command  or  Service  response 
could  be  formulated  in  question  6.  Finally,  ten  of  the  40  commands  returning  surveys  had 
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some  PCOs  that  were  notified  of  estimating  system  problems  and  some  not.  Examining  each 
bit  of  information  independently  does  not  allow  for  a  specific  conclusion  to  be  formed. 
However,  when  analyzed  together,  they  indicate  that  the  Government  could  be  taking  a  more 
coordinated  effort  to  ensure  a  fair  and  reasonable  contract  price  is  reached. 

C.         RECOMMENDATIONS 

This  section  discusses  the  recommendations  concluded  from  this  research  effort. 

1.  DCAA  should  provide  all  buying  commands  with  an  annual  listing  of 
contractors  evaluated  as  high  risk,  based  on  DCAA's  Internal  Control  Audit  Planning 
Summary  (ICAPS)  System. 

Providing  this  kind  of  information  would  offer  PCOs  more  data  upon  which  to 
determine  if  additional  cost  analysis  should  be  conducted.  Using  these  data,  PCOs  could 
evaluate  the  resources  available  and  time  limits  of  each  procurement  against  the  risk  of  an 
inaccurate  contractor  proposal.  Resources  would  be  more  focused  on  the  proposals  that  truly 
represent  a  risk  to  the  Government. 

2.  Buying  commands  management  and  contracting  directorates  should  act  as 
a  conduit  for  information  regarding  contractors  estimating  systems. 

PCOs  have  identified  deficient  contractor  systems  to  their  chain  of  commands  but 
none  of  the  PCOs  identified  their  chain  of  command  as  a  source  of  information  about  these 
systems.  Encouraging  the  management  structure  to  act  as  a  conduit  and  coordinate  a  more 
consolidated  and  synergistic  response  toward  a  specific  contractor  could  be  achieved.  The 
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contractor  would  have  more  impetus  to  correct  the  deficiencies  or  seek  approval  of  his 
estimating  system.  By  reducing  the  number  of  contractors  with  problem  systems,  savings 
can  be  achieved  by  decreasing  ACO  and  DCAA  workloads. 

3.  PCOs  should  be  encouraged  not  to  award  to  firms  with  deficient  or 
disapproved  estimating  systems. 

As  seen  from  the  research,  PCOs  use  actions  that  lengthen  the  contract  process  and 
expend  additional  resources  when  dealing  with  contractors  that  have  problem  systems.  The 
Army  Contracting  Officer  example  in  Chapter  III  is  the  perfect  case.  He  submitted 
approximately  1 00  requests  to  the  contractor  for  additional  information.  This  was  more  than 
twice  the  normal  number.  It  takes  valuable  resources  from  both  the  Government  and  the 
contractor  to  generate,  answer  and  evaluate  these  requests.  By  not  awarding  to  these 
contractors,  the  Government  saves  time,  saves  resources,  and  sends  a  strong  message  to 
contractors. 

D.         SUMMARY  OF  RESEARCH  QUESTIONS 

In  order  to  accomplish  the  objectives  of  this  study,  the  following  research  questions 
were  pursued: 

1.  Primary  research  question.  What  actions  does  the  Procuring  Contracting 
Officer  take  when  notified  that  a  contractor's  estimating  system  has  been  disapproved 
by  the  Administrative  Contracting  Officer  or  has  deficiencies? 
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In  general,  PCOs  take  actions  internal  to  the  Government  when  notified  of  contractor 
estimating  systems  being  disapproved  or  having  deficiencies.  These  actions  include 
determining  the  impact  of  the  disapproval  or  deficiency,  holding  discussions  with  the  ACO 
and  DCAA  auditor,  and  informing  their  management  structure.  In  the  surveyed  population, 
PCOs  took  actions  internal  to  the  Government  62%  of  the  time.  When  specifically  in  the 
negotiation  phase  of  a  procurement,  PCOs  use  a  combination  of  additional  cost  analysis  and 
allowing  the  contractor  additional  time  to  correct  the  problem  and  resubmit  a  corrected 
proposal.  These  actions  were  taken  56%  of  the  time  for  the  sample  population. 

2.  Subsidiary  research  question  #1.  Are  the  alternative  actions  specified  in 
DFARS  215.81  l-70(g)(2)  adequate  to  motivate  a  contractor  to  respond  to  the  PCO's 
desires? 

Yes,  the  alternative  actions  in  the  DFARS  are  adequate  to  motivate  contractors.  The 
survey  shows  that  the  majority  of  PCOs  believed  that  the  contractor  was  motivated  by  the 
action  they  took.  Many  PCOs  expressed  contractors  are  concerned  about  the  impact  their 
deficient  estimating  system  has  on  current  and  future  business  therefore  they  respond  to  the 
PCO's  actions. 

3.  Subsidiary  research  question  #2.  Are  there  more  effective  actions  currently 
being  used  by  PCOs  and  if  so  should  they  be  incorporated  into  the  DFARS? 

There  is  one  action  currently  being  used  by  PCOs  that  is  not  listed  in  the  DFARS. 
This  action  is  to  award  the  contract  to  the  next  competitor  at  a  presumably  higher  contract 
price.  It  cannot  be  determined  if  this  action  is  more  effective  than  current  DFARS  actions 
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but  the  responding  PCOs  stated  they  believe  it  is  effective.  In  the  opinion  of  the  researcher, 
this  option  is  a  good  one  but  should  not  be  included  in  the  DFARS.  By  putting  this  option 
in  the  DFARS  some  PCOs  will  take  it  as  a  required  action.  That  is  not  the  intent.  The  intent 
is  that  it  should  be  used  if  the  situation  warrants. 

4.  Subsidiary  research  question  #3.  If  the  PCO  gathers  independent  data 
indicating  a  contractor  has  estimating  system  deficiencies,  how  does  this  affect  the 
performance  of  his  or  her  duties? 

Only  two  PCOs  identified  they  had  gathered  independent  data  about  contractors' 
estimating  systems.  Insufficient  data  were  provided  to  be  able  to  evaluate  the  affects  on 
PCOs  actions.  It  is  unclear  whether  this  was  caused  by  a  fault  in  the  questionnaire  or  few 
PCOs  gather  independent  data  about  contractor  estimating  systems. 

E.         ADDITIONAL  AREAS  OF  RESEARCH 

As  a  result  of  the  research  conducted  on  contractors'  estimating  systems,  the 
following  areas  warrant  further  investigation: 

1 .  Survey  buying  command  management  structures  regarding  information  flow  in 
regards  to  deficient  estimating  systems  within  the  command. 

2.  Survey  Administrative  Contacting  Offices  regarding  affect  of  PCO  actions  on  a 
contractor. 

3.  Analyze  DCMC  headquarters  current  oversight  of  regional  offices  in  terms  of 
reporting  deficiencies  and  disapprovals  and  adequacy  of  follow  up  reporting  systems. 


4.  Survey  industry  regarding  whether  the  current  estimating  system  regulations  and 
oversight  structure  are  too  burdensome  and  if  industry  can  suggest  alternatives. 

5.  Analysis  ACO  adequacy  determinationdecisions  of  contractor  estimating  systems. 
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APPENDIX  A.  PCO  SURVEY 


Directions:  Please  complete  the  following  information. 

1 .  How  many  years  of  contracting  experience  do  you  have? 

a.  [    ]0-3         b.  [    ]4-8         c.  [    ]  9-15       d.  [    ]  16-25     e.  [    ]  26  or  greater 

2.  What  is  the  average  dollar  value  of  contracts  you  are  responsible  for  per  year? 
a.  [    ]  041,000,000  b.  [    ]  $1,000,000-$  10,000,000 

c.  [    ]$10,000,000-$50,000,000         d.  [    ]  $50,000,000-$  100,000,000 
e.  [    ]  $100,000,000  or  greater 

3.  Have  you  ever  received  notification  of  a  contractor's  estimating  system  having 
deficiencies  or  being  disapproved? 

a.  [    ]  yes         b.  [    ]  no 

4.  Why  were  you  notified?  


If  yes,  please  discuss  the  last  three  occurrences: 

5.  How  did  you  learn  of  the  #1 

deficiencies  or  disapproval? 


#2 


#3 


ACO 
DCAA 
Contractor 
Cost  Analyst 
Govt.  Tech/Eng 
Other  (specify) 


6.  What  action  did  you  take  once  notified? 


#1 


#2 


#3 
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7.  If  in  negotiations,  how  did  you  proceed? 

a.  No  change  in  plan. 

b.  Allowed  the  contractor  additional  time  to  correct  the 
deficiency  and  submit  a  corrected  proposal. 

c.  Considered  another  type  of  contract. 

d.  Used  additional  cost  analysis  to  determine  the 
reasonableness  of  the  cost  elements  affected  by  the  system's 
deficiency. 

e.  Segregated  the  questionable  areas  as  a  cost  reimbursable 
line  item. 

f.  Reduced  the  negotiation  objective  for  profit  or  fee. 

g.  Included  a  contract  reopener  clause  that  provides  for 
adjustment  of  the  contract  amount  after  award. 


#1 


#2 


#3 


h.  Other  (specify) 

#1  


#2 


#3 


8.  Did  the  actions  you  took  motivate  the  contractor  to  give  the  Government  a  fair  and 
reasonable  price? 

a.  [    ]  yes        b.  [    ]  no 


9.  Why? 


Optional:  If  you  have  any  questions  about  the  survey  or  want  to  discuss  any  of  your 
answers  please  print  your  name  and  telephone  or  E-mail  address  and  I  will  contact  you. 
Name:  


Phone  number  or  E-mail  address: 


Thank  you  for  your  participation! 
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APPENDIX  B.  SURVEY  RESPONSES 

The  following  tables  are  the  aggregate  responses  received  from  the  researchers  survey. 
Question  1.  How  many  years  of  contracting  experience  do  you  have? 


Command 

la 

lb 

lc 

Id 

le 

Advanced  Research  Projects  Agency 

0 

0 

0 

1 

0 

Aeronautical  Systems  Center 

0 

0 

0 

2 

1 

Air  Force  Development  Test  Center 

0 

0 

0 

1 

0 

Air  Force  Material  Command 

0 

0 

1 

1 

0 

Air  Force  Office  of  Scientific  Research 

0 

0 

0 

1 

0 

Airborne  Warning  &  Control  Systems 

0 

0 

0 

1 

0 

Area  Attack  SPO 

0 

0 

0 

1 

0 

Armament  Research,  Development  &  Engineering  Center 

0 

0 

0 

3 

0 

Arnold  Engineering  Development  Center 

0 

0 

0 

1 

0 

Aviation  &  Troop  Command 

0 

0 

2 

3 

1 

B-2  Program  Office 

0 

0 

0 

1 

0 

Chemical  &  Biological  Defense  Command 

0 

0 

0 

0 

1 

Cheyenne  Mountain  Complex  System 

0 

0 

0 

0 

1 

Communications  Electronics  Command 

0 

0 

0 

4 

0 

DOD  Ballistic  Missile  Defense  Organization 

0 

0 

0 

1 

0 

Electronic  Systems  Center 

0 

0 

1 

2 

0 

F- 16  SPO 

0 

0 

0 

0 

1 

F-22  System  Program  Office 

0 

0 

0 

0 

1 

Joint  Stars 

0 

0 

0 

1 

0 

Joint  Tactical  Information  Distribution  System 

0 

0 

0 

1 

0 

Marine  Corps  Systems  Command 

0 

0 

0 

1 

0 

Missile  Command 

0 

1 

1 

9 

3 
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Command 

la 

lb 

lc 

Id 

le 

Naval  Air  Systems  Command 

1 

0 

5 

6 

1 

Naval  Research  Laboratory 

0 

0 

0 

1 

0 

Naval  Sea  Systems  Command 

0 

1 

4 

8 

2 

Naval  Surface  Warfare  Center 

0 

0 

0 

1 

1 

Naval  Undersea  Warfare  Center 

0 

0 

0 

0 

1 

Navy  Inventory  Control  Point 

0 

0 

0 

0 

1 

Office  of  Naval  Research 

0 

0 

2 

0 

1 

Ogden  Air  Logistics  Center 

0 

0 

0 

0 

1 

Phillips  Laboratory 

0 

1 

0 

1 

0 

San  Antonio  Air  Logistics  Center 

1 

0 

1 

0 

0 

Space  &  Missile  Systems  Center 

0 

0 

1 

3 

0 

Space  and  Naval  Warfare  Systems  Command 

0 

0 

0 

2 

1 

Space  &  Strategic  Defense  Command 

0 

0 

0 

1 

0 

Strategic  Systems  Programs 

0 

0 

0 

1 

0 

US  Army  Space  Command 

0 

0 

0 

0 

1 

Utility  Helicopters 

0 

0 

1 

0 

0 

Warner  Robins  Air  Logistics  Center 

0 

0 

1 

2 

0 

Wright  Laboratories 

0 

0 

1 

0 

0 
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Question  2.  What  is  the  average  dollar  value  of  contracts  you  are  responsible  for  per 
year? 


Command 

2a 

2b 

2c 

2d 

2e 

Advanced  Research  Projects  Agency 

0 

0 

1 

0 

0 

Aeronautical  Systems  Center 

0 

0 

0 

1 

2 

Air  Force  Development  Test  Center 

0 

0 

0 

0 

1 

Air  Force  Material  Command 

0 

1 

0 

1 

0 

Air  Force  Office  of  Scientific  Research 

1 

0 

0 

0 

0 

Airborne  Warning  &  Control  Systems 

0 

0 

0 

0 

1 

Area  Attack  SPO 

0 

0 

1 

0 

0 

Armament  Research,  Development  &  Engineering  Center 

0 

0 

1 

1 

1 

Arnold  Engineering  Development  Center 

0 

0 

0 

0 

1 

Aviation  &  Troop  Command 

0 

0 

1 

3 

2 

B-2  Program  Office 

0 

0 

0 

0 

1 

Chemical  &  Biological  Defense  Command 

0 

1 

0 

0 

0 

Cheyenne  Mountain  Complex  System 

0 

0 

0 

0 

1 

Communications  Electronics  Command 

0 

0 

0 

2 

2 

DOD  Ballistic  Missile  Defense  Organization 

0 

0 

1 

0 

0 

Electronic  Systems  Center 

0 

0 

2 

0 

1 

F-16SPO 

0 

0 

0 

1 

0 

F-22  System  Program  Office 

0 

0 

0 

0 

1 

Joint  Stars 

0 

0 

0 

0 

1 

Joint  Tactical  Information  Distribution  System 

0 

0 

1 

0 

0 

Marine  Corps  Systems  Command 

0 

0 

0 

0 

1 

Missile  Command 

2 

2 

2 

2 

6 

Naval  Air  Systems  Command 

0 

0 

2 

4 

7 
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Command 

2a 

2b 

2c 

2d 

2e 

Naval  Research  Laboratory 

0 

0 

0 

0 

1 

Naval  Sea  Systems  Command 

0 

2 

2 

1 

10 

Naval  Surface  Warfare  Center 

0 

0 

1 

0 

1 

Naval  Undersea  Warfare  Center 

0 

1 

0 

0 

0 

Navy  Inventory  Control  Point 

1 

0 

0 

0 

0 

Office  of  Naval  Research 

0 

1 

0 

0 

2 

Ogden  Air  Logistics  Center 

1 

0 

0 

0 

0 

Phillips  Laboratory 

0 

0 

1 

1 

0 

San  Antonio  Air  Logistics  Center 

1 

1 

0 

0 

0 

Space  &  Missile  Systems  Center 

0 

0 

0 

0 

4 

Space  and  Naval  Warfare  Systems  Command 

0 

0 

0 

1 

2 

Space  &  Strategic  Defense  Command 

0 

0 

1 

0 

0 

Strategic  Systems  Programs 

0 

0 

1 

0 

0 

US  Army  Space  Command 

0 

0 

1 

0 

0 

Utility  Helicopters 

0 

0 

0 

0 

1 

Warner  Robins  Air  Logistics  Center 

1 

1 

0 

0 

1 

Wright  Laboratories 

0 

0 

1 

0 

0 
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Question  3.  Have  you  ever  received  notification  of  a  contractors  estimating  system 
having  deficiencies  or  being  disapproved? 


Command 

3a 

3b 

Advanced  Research  Projects  Agency 

0 

1 

Aeronautical  Systems  Center 

2 

1 

Air  Force  Development  Test  Center 

0 

1 

Air  Force  Material  Command 

2 

0 

Air  Force  Office  of  Scientific  Research 

1 

0 

Airborne  Warning  &  Control  Systems 

1 

0 

Area  Attack  SPO 

1 

0 

Armament  Research,  Development  &  Engineering  Center 

1 

2 

Arnold  Engineering  Development  Center 

0 

1 

Aviation  &  Troop  Command 

4 

2 

B-2  Program  Office 

0 

1 

Chemical  &  Biological  Defense  Command 

0 

Cheyenne  Mountain  Complex  System 

0 

Communications  Electronics  Command 

4 

0 

DOD  Ballistic  Missile  Defense  Organization 

0 

Electronic  Systems  Center 

2 

F-16SPO 

0 

F-22  System  Program  Office 

0 

Joint  Stars 

0 

Joint  Tactical  Information  Distribution  System 

0 

Marine  Corps  Systems  Command 

0 

Missile  Command 

12 

2 

Naval  Air  Systems  Command 

8 

5 
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Command 

3a 

3b 

Naval  Research  Laboratory 

1 

0 

Naval  Sea  Systems  Command 

12 

3 

Naval  Surface  Warfare  Center 

2 

0 

Naval  Undersea  Warfare  Center 

0 

Navy  Inventory  Control  Point 

0 

Office  of  Naval  Research 

0 

3 

Ogden  Air  Logistics  Center 

0 

Phillips  Laboratory 

1 

San  Antonio  Air  Logistics  Center 

1 

Space  &  Missile  Systems  Center 

3 

Space  and  Naval  Warfare  Systems  Command 

2 

Space  &  Strategic  Defense  Command 

0 

Strategic  Systems  Programs 

0 

US  Army  Space  Command 

0 

1 

Utility  Helicopters 

0 

1 

Warner  Robins  Air  Logistics  Center 

3 

0 

Wright  Laboratories 

1 

0 
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Question  4.  Why  were  you  notified? 


Command 

1 

2 

3 

4 

Advanced  Research  Projects  Agency 

0 

0 

0 

0 

Aeronautical  Systems  Center 

0 

1 

1 

0 

Air  Force  Development  Test  Center 

0 

0 

0 

0 

Air  Force  Material  Command 

1 

0 

0 

1 

Air  Force  Office  of  Scientific  Research 

1 

0 

0 

0 

Airborne  Warning  &  Control  Systems 

0 

1 

0 

0 

Area  Attack  SPO 

0 

0 

0 

1 

Armament  Research,  Development  &  Engineering  Center 

1 

0 

0 

0 

Arnold  Engineering  Development  Center 

0 

0 

0 

0 

Aviation  &  Troop  Command 

0 

4 

0 

0 

B-2  Program  Office 

0 

0 

0 

0 

Chemical  &  Biological  Defense  Command 

1 

0 

0 

0 

Cheyenne  Mountain  Complex  System 

1 

0 

0 

0 

Communications  Electronics  Command 

3 

1 

0 

0 

DOD  Ballistic  Missile  Defense  Organization 

1 

0 

0 

0 

Electronic  Systems  Center 

0 

1 

0 

0 

F-16  SPO 

0 

1 

0 

0 

F-22  System  Program  Office 

1 

0 

0 

0 

Joint  Stars 

0 

1 

0 

0 

Joint  Tactical  Information  Distribution  System 

1 

0 

0 

0 

Marine  Corps  Systems  Command 

0 

1 

0 

0 

Missile  Command 

8 

4 

0 

0 

Naval  Air  Systems  Command 

2 

2 

2 

2 

Naval  Research  Laboratory 

0 

1 

0 

0 
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Command 

1 

2 

3 

4 

Naval  Sea  Systems  Command 

6 

6 

0 

0 

Naval  Surface  Warfare  Center 

2 

0 

0 

0 

Naval  Undersea  Warfare  Center 

1 

0 

0 

0 

Navy  Inventory  Control  Point 

0 

0 

1 

0 

Office  of  Naval  Research 

0 

0 

0 

0 

Ogden  Air  Logistics  Center 

0 

1 

0 

0 

Phillips  Laboratory 

1 

0 

0 

0 

San  Antonio  Air  Logistics  Center 

1 

0 

0 

0 

Space  &  Missile  Systems  Center 

0 

1 

0 

0 

Space  and  Naval  Warfare  Systems  Command 

1 

0 

0 

0 

Space  &  Strategic  Defense  Command 

0 

0 

0 

1 

Strategic  Systems  Programs 

1 

0 

0 

0 

US  Army  Space  Command 

0 

0 

0 

0 

Utility  Helicopters 

0 

0 

0 

0 

Warner  Robins  Air  Logistics  Center 

2 

1 

0 

0 

Wright  Laboratories 

1 

0 

0 

0 

Note: 

1 .  Result  of  a  proposal  evaluation  request. 

2.  PCO  on  standard  distribution  list  or  PCO  of  record. 

3.  Miscellaneous. 

4.  No  response 
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Question  5.  How  did  you  learn  of  the  deficiencies  or  disapproval? 


Command 

5a 

5b 

5c 

5d 

5e 

5f 

Advanced  Research  Projects  Agency 

0 

0 

0 

0 

0 

0 

Aeronautical  Systems  Center 

4 

0 

0 

0 

0 

0 

Air  Force  Development  Test  Center 

0 

0 

0 

0 

0 

0 

Air  Force  Material  Command 

2 

0 

1 

0 

0 

Air  Force  Office  of  Scientific  Research 

1 

0 

0 

0 

0 

Airborne  Warning  &  Control  Systems 

1 

0 

0 

0 

0 

Area  Attack  SPO 

1 

0 

0 

0 

0 

Armament  Research,  Development  &  Engineering  Center 

1 

0 

0 

0 

0 

Arnold  Engineering  Development  Center 

0 

0 

0 

0 

0 

0 

Aviation  &  Troop  Command 

3 

2 

0 

0 

1 

0 

B-2  Program  Office 

0 

0 

0 

0 

0 

0 

Chemical  &  Biological  Defense  Command 

0 

3 

0 

0 

0 

0 

Cheyenne  Mountain  Complex  System 

0 

3 

0 

0 

0 

0 

Communications  Electronics  Command 

4 

6 

0 

0 

0 

0 

DOD  Ballistic  Missile  Defense  Organization 

0 

1 

0 

0 

0 

0 

Electronic  Systems  Center 

1 

0 

0 

0 

0 

0 

F-16  SPO 

2 

0 

0 

0 

0 

0 

F-22  System  Program  Office 

1 

2 

0 

0 

0 

0 

Joint  Stars 

1 

0 

0 

0 

0 

0 

Joint  Tactical  Information  Distribution  System 

3 

3 

0 

0 

0 

0 

Marine  Corps  Systems  Command 

1 

0 

0 

0 

0 

1 

Missile  Command 

6 

21 

3 

0 

0 

0 

Naval  Air  Systems  Command 

10 

6 

1 

1 

0 

0 

Naval  Research  Laboratory 

0 

1 

0 

0 

0 

0 
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Command 

5a 

5b 

5c 

5d 

5e 

5f 

Naval  Sea  Systems  Command 

17 

15 

1 

0 

0 

0 

Naval  Surface  Warfare  Center 

3 

3 

1 

0 

0 

0 

Naval  Undersea  Warfare  Center 

0 

1 

0 

0 

0 

0 

Navy  Inventory  Control  Point 

1 

0 

1 

0 

0 

0 

Office  of  Naval  Research 

0 

0 

0 

0 

0 

0 

Ogden  Air  Logistics  Center 

0 

3 

0 

0 

0 

0 

Phillips  Laboratory 

1 

1 

0 

0 

0 

0 

San  Antonio  Air  Logistics  Center 

0 

3 

0 

0 

0 

0 

Space  &  Missile  Systems  Center 

0 

0 

0 

0 

0 

0 

Space  and  Naval  Warfare  Systems  Command 

1 

1 

0 

0 

0 

0 

Space  &  Strategic  Defense  Command 

0 

1 

0 

0 

0 

0 

Strategic  Systems  Programs 

0 

3 

0 

0 

0 

0 

US  Army  Space  Command 

0 

0 

0 

0 

0 

0 

Utility  Helicopters 

0 

0 

0 

0 

0 

0 

Warner  Robins  Air  Logistics  Center 

7 

3 

3 

3 

0 

0 

Wright  Laboratories 

2 

1 

0 

0 

0 

0 
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Question  6.  What  actions  did  you  take  once  notified? 


Command 

1 

2 

3 

4 

Advanced  Research  Projects  Agency 

0 

0 

0 

0 

Aeronautical  Systems  Center 

0 

3 

3 

0 

Air  Force  Development  Test  Center 

0 

0 

0 

0 

Air  Force  Material  Command 

6 

0 

0 

0 

Air  Force  Office  of  Scientific  Research 

1 

0 

1 

0 

Airborne  Warning  &  Control  Systems 

0 

2 

0 

0 

Area  Attack  SPO 

1 

0 

0 

0 

Armament  Research,  Development  &  Engineering  Center 

0 

2 

0 

0 

Arnold  Engineering  Development  Center 

0 

0 

0 

0 

Aviation  &  Troop  Command 

3 

3 

0 

0 

B-2  Program  Office 

0 

0 

0 

0 

Chemical  &  Biological  Defense  Command 

2 

0 

1 

0 

Cheyenne  Mountain  Complex  System 

3 

0 

0 

0 

Communications  Electronics  Command 

1 

2 

5 

0 

DOD  Ballistic  Missile  Defense  Organization 

1 

0 

0 

0 

Electronic  Systems  Center 

0 

0 

1 

0 

F- 16  SPO 

0 

2 

0 

0 

F-22  System  Program  Office 

3 

0 

0 

0 

Joint  Stars 

0 

0 

2 

0 

Joint  Tactical  Information  Distribution  System 

3 

0 

0 

0 

Marine  Corps  Systems  Command 

3 

0 

0 

0 

Missile  Command 

22 

1 

3 

2 

Naval  Air  Systems  Command 

8 

2 

6 

2 

Naval  Research  Laboratory 

1 

0 

0 

0 
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Command 

1 

2 

3 

4 

Naval  Sea  Systems  Command 

12 

5 

10 

0 

Naval  Surface  Warfare  Center 

2 

2 

0 

1 

Naval  Undersea  Warfare  Center 

1 

0 

0 

0 

Navy  Inventory  Control  Point 

1 

0 

0 

0 

Office  of  Naval  Research 

0 

0 

0 

0 

Ogden  Air  Logistics  Center 

3 

0 

0 

0 

Phillips  Laboratory 

2 

0 

0 

0 

San  Antonio  Air  Logistics  Center 

2 

1 

0 

0 

Space  &  Missile  Systems  Center 

2 

0 

0 

0 

Space  and  Naval  Warfare  Systems  Command 

0 

0 

2 

0 

Space  &  Strategic  Defense  Command 

0 

1 

0 

0 

Strategic  Systems  Programs 

3 

0 

0 

0 

US  Army  Space  Command 

0 

0 

0 

0 

Utility  Helicopters 

0 

0 

0 

0 

Warner  Robins  Air  Logistics  Center 

4 

0 

0 

1 

Wright  Laboratories 

3 

0 

0 

0 

Note: 

1 .  Action  internal  to  the  Government. 

2.  Action  directly  with  the  contractor. 

3.  Action  including  the  contractor  and  the  Government. 

4.  Miscellaneous. 
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Question  7.  If  in  negotiations  how  did  you  proceed? 


Command 

7a 

7b 

7c 

7d 

7e 

7f 

7g 

7h 

Advanced  Research  Projects  Agency 

0 

0 

0 

0 

0 

0 

0 

0 

Aeronautical  Systems  Center 

1 

1 

0 

1 

1 

0 

1 

0 

Air  Force  Development  Test  Center 

0 

0 

0 

0 

0 

0 

0 

0 

Air  Force  Material  Command 

0 

2 

0 

2 

0 

0 

0 

0 

Air  Force  Office  of  Scientific  Research 

0 

1 

0 

1 

0 

0 

0 

0 

Airborne  Warning  &  Control  Systems 

0 

1 

0 

0 

0 

0 

0 

0 

Area  Attack  SPO 

0 

0 

0 

1 

0 

0 

0 

0 

Armament  Research,  Development  & 
Engineering  Center 

0 

1 

0 

1 

0 

1 

0 

0 

Arnold  Engineering  Development  Center 

0 

0 

0 

0 

0 

0 

0 

0 

Aviation  &  Troop  Command 

0 

2 

0 

4 

0 

0 

1 

0 

B-2  Program  Office 

0 

0 

0 

0 

0 

0 

0 

0 

Chemical  &  Biological  Defense  Command 

0 

1 

0 

2 

0 

0 

0 

0 

Cheyenne  Mountain  Complex  System 

3 

0 

0 

0 

0 

0 

0 

0 

Communications  Electronics  Command 

0 

5 

1 

2 

1 

0 

2 

0 

DOD  Ballistic  Missile  Defense  Organization 

0 

0 

0 

1 

0 

0 

1 

0 

Electronic  Systems  Center 

0 

0 

0 

0 

0 

0 

0 

0 

F-16  SPO 

0 

2 

0 

0 

0 

0 

0 

0 

F-22  System  Program  Office 

0 

1 

0 

1 

0 

0 

0 

1 

Joint  Stars 

0 

1 

1 

1 

0 

0 

0 

0 

Joint  Tactical  Information  Distribution  System 

1 

0 

0 

0 

0 

0 

0 

0 

Marine  Corps  Systems  Command 

0 

0 

0 

1 

0 

0 

0 

0 

Missile  Command 

11 

5 

0 

11 

0 

0 

6 

1 

Naval  Air  Systems  Command 

2 

3 

0 

6 

0 

6 

1 

4 

Naval  Research  Laboratory 

0 

3 

0 

0 

0 

0 

0 

0 
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Command 

7a 

7b 

7c 

7d 

7e 

7f 

Vg 

7h 

Naval  Sea  Systems  Command 

4 

3 

0 

7 

0 

1 

6 

2 

Naval  Surface  Warfare  Center 

0 

3 

1 

3 

2 

2 

0 

0 

Naval  Undersea  Warfare  Center 

1 

0 

0 

1 

0 

1 

0 

0 

Navy  Inventory  Control  Point 

0 

1 

0 

1 

0 

0 

1 

0 

Office  of  Naval  Research 

0 

0 

0 

0 

0 

0 

0 

0 

Ogden  Air  Logistics  Center 

0 

0 

1 

0 

1 

0 

1 

0 

Phillips  Laboratory 

0 

0 

1 

0 

0 

1 

1 

0 

San  Antonio  Air  Logistics  Center 

0 

3 

0 

3 

0 

0 

0 

0 

Space  &  Missile  Systems  Center 

0 

0 

0 

0 

0 

0 

0 

2 

Space  and  Naval  Warfare  Systems  Command 

0 

2 

0 

0 

0 

0 

0 

0 

Space  &  Strategic  Defense  Command 

0 

1 

0 

1 

0 

0 

0 

0 

Strategic  Systems  Programs 

0 

0 

0 

2 

0 

0 

1 

0 

US  Army  Space  Command 

0 

0 

0 

0 

0 

0 

0 

0 

Utility  Helicopters 

0 

0 

0 

0 

0 

0 

0 

0 

Warner  Robins  Air  Logistics  Center 

0 

4 

0 

1 

0 

1 

0 

0 

Wright  Laboratories 

1 

0 

0 

1 

1 

0 

0 

0 
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Question  8.  Did  the  actions  you  took  motivate  the  contractor  to  give  the  Government  a 
fair  and  reasonable  price? 


Command 

yes 

no 

Advanced  Research  Projects  Agency 

0 

0 

Aeronautical  Systems  Center 

2 

0 

Air  Force  Development  Test  Center 

0 

0 

Air  Force  Material  Command 

1 

0 

Air  Force  Office  of  Scientific  Research 

1 

0 

Airborne  Warning  &  Control  Systems 

1 

0 

Area  Attack  SPO 

0 

0 

Armament  Research,  Development  &  Engineering  Center 

1 

0 

Arnold  Engineering  Development  Center 

0 

0 

Aviation  &  Troop  Command 

3 

1 

B-2  Program  Office 

0 

0 

Chemical  &  Biological  Defense  Command 

0 

1 

Cheyenne  Mountain  Complex  System 

1 

0 

Communications  Electronics  Command 

2 

0 

DOD  Ballistic  Missile  Defense  Organization 

1 

0 

Electronic  Systems  Center 

0 

0 

F- 16  SPO 

1 

0 

F-22  System  Program  Office 

1 

0 

Joint  Stars 

1 

0 

Joint  Tactical  Information  Distribution  System 

1 

0 

Marine  Corps  Systems  Command 

0 

0 

Missile  Command 

7 

2 

Naval  Air  Systems  Command 

4 

3 
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Command 

yes 

no 

Naval  Research  Laboratory 

1 

0 

Naval  Sea  Systems  Command 

4 

4 

Naval  Surface  Warfare  Center 

2 

0 

Naval  Undersea  Warfare  Center 

1 

0 

Navy  Inventory  Control  Point 

0 

1 

Office  of  Naval  Research 

0 

0 

Ogden  Air  Logistics  Center 

0 

1 

Phillips  Laboratory 

0 

1 

San  Antonio  Air  Logistics  Center 

1 

0 

Space  &  Missile  Systems  Center 

1 

0 

Space  and  Naval  Warfare  Systems  Command 

0 

0 

Space  &  Strategic  Defense  Command 

1 

0 

Strategic  Systems  Programs 

1 

0 

US  Army  Space  Command 

0 

0 

Utility  Helicopters 

0 

0 

Warner  Robins  Air  Logistics  Center 

2 

0 

Wright  Laboratories 

1 

0 
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2.  Dudley  Knox  Library  2 
Naval  Postgraduate  School 

411  DyerRd. 

Monterey,  CA  93943-5101 

3.  Defense  Logistics  Studies  Information  Exchange  1 
U.S.  Army  Logistics  Management  College 

Fort  Lee,  Virginia  22134-5027 

4.  CDR  Danny  A.  Shockley  (Code  SM/Sb)  1 
Naval  Postgraduate  School 

Monterey,  CA  93943-5103 

5.  Dr.  David  V.  Lamm  (Code  SM/Lt)  5 
Naval  Postgraduate  School 

Monterey,  CA  93943-5103 

6.  Mr.  Dave  Ricci  1 
Defense  Logistics  Agency 

8725  John  J  Kingman  Road  Suite  2533 
Ft.  Belvor,  VA  22060-6221 
Attn:  AQOD/Dave  Ricci 
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